Struct Interpreter 

pub struct Interpreter { /* private fields */ }

The Interpreter holds the model and manages sessions.

It’s internals use a shared mutex so it’s thread-safe. net data holder. multiple sessions could share same net.

Implementations

impl Interpreter

pub fn from_file(path: impl AsRef<Path>) -> Result<Self>

Create an net/interpreter from a file.

path: the file path of the model

return: the created net/interpreter

pub fn from_bytes(bytes: impl AsRef<[u8]>) -> Result<Self>

Create an net/interpreter from a buffer.

bytes: the buffer of the model

return: the created net/interpreter

pub fn set_session_mode(&self, mode: SessionMode)

Set session mode

mode: the session mode

Warning: It should be called before create session!

pub fn resize_session(&self, session: &mut Session<'_>)

call this function to get tensors ready.

output tensor buffer (host or deviceId) should be retrieved after resize of any input tensor.

session: the session to be prepared

pub fn resize_session_reallocate(&self, session: &mut Session<'_>)

Resize session and reallocate the buffer.

session: the session to be prepared.

Note

NeedRelloc is default to 1, 1 means need realloc!

pub fn resize_tensor<H: HalideType, M: TensorMachine>( &self, tensor: &mut TensorRef<H, M>, dims: impl AsTensorShape, )

Resize the tensor using the given shape

pub fn resize_tensor_by_name( &self, session: &mut Session<'_>, name: impl AsRef<str>, dims: impl AsTensorShape, ) -> Result<()>

Resize the tenror by name using the given shape

pub fn resize_tensor_by_nchw<H: HalideType, M: TensorMachine>( &self, tensor: &mut TensorRef<H, M>, batch: u16, channel: u16, height: u16, width: u16, )

Resize tensor by

• N -> batch
• C -> channel
• H -> height
• W -> width

pub fn create_session(&self, schedule: ScheduleConfig) -> Result<Session<'_>>

Create a session with session config. Session will be managed in net/interpreter.

schedule : the config of the session

return: the created session

pub unsafe fn release_model(&mut self)

Release the model file buffer

Safety

This function is marked unsafe since it’s not clear what the safety guarantees are right now. With a simple test it caused a segfault so it’s marked unsafe

pub fn create_multipath_session( &self, schedule: impl IntoIterator<Item = ScheduleConfig>, ) -> Result<Session<'_>>

Create multi-path session with schedule configs and user-specified runtime. created session will be managed in net/interpreter.

schedule : the config of the session

return: the created session

pub fn model_print_io(path: impl AsRef<Path>) -> Result<()>

Print all input and output tensors info.

pub fn inputs<'i>(&self, session: &'i mut Session<'_>) -> TensorListMut<'i>

Get the input tensor of the session.

session: the session to get input tensor

return: List of input tensors

pub fn input<'s, H: HalideType>( &self, session: &'s mut Session<'_>, name: impl AsRef<str>, ) -> Result<&'s mut TensorRef<H, Device>>

Get the input tensor of the session by name.

session: the session to get input tensor from

name: the name of the input tensor

return: the input tensor

pub fn raw_input<'s>( &self, session: &'s mut Session<'_>, name: impl AsRef<str>, ) -> Result<&'s AnyTensorRef>

Get the raw input tensor of a session by name

pub unsafe fn input_unresized<'s, H: HalideType>( &self, session: &'s mut Session<'_>, name: impl AsRef<str>, ) -> Result<&'s mut TensorRef<H, Device>>

Safety

Warning We Still don’t know the safety guarantees of this function so it’s marked unsafe

pub unsafe fn input_unchecked<'s, H: HalideType>( &self, session: &'s mut Session<'_>, name: impl AsRef<str>, ) -> &'s mut TensorRef<H, Device>

Safety

Very unsafe since it doesn’t check the type of the tensor as well as the shape of the tensor

Panics if the name is not ascii Undefined Behavior if the tensor is not of type H

pub fn output<'s, H: HalideType>( &self, session: &'s Session<'_>, name: impl AsRef<str>, ) -> Result<&'s TensorRef<H, Device>>

Get the output tensor of a session by name

session : the session to get output tensor from

name : the name of the output tensor

pub fn raw_output<'s>( &self, session: &'s Session<'_>, name: impl AsRef<str>, ) -> Result<&'s AnyTensorRef>

Get the raw output tensor of a session by name

pub fn run_session(&self, session: &Session<'_>) -> Result<()>

Run a session

pub fn run_session_with_callback( &self, session: &Session<'_>, before: impl Fn(&[&AnyTensorRef], OperatorInfo<'_>) -> bool + 'static, end: impl Fn(&[&AnyTensorRef], OperatorInfo<'_>) -> bool + 'static, sync: bool, ) -> Result<()>

Run a session with a callback

session : the session to run

before : a callback before each op. return true to run the op; return false to skip the op.

after : a callback after each op. return true to continue running; return false to interrupt the session.

sync : synchronously wait for finish of execution or not.

pub fn outputs<'o>(&self, session: &'o Session<'_>) -> TensorList<'o>

Get all output tensors of a session

pub fn set_cache_file( &self, path: impl AsRef<Path>, key_size: usize, ) -> Result<()>

If the cache exist, try to load cache from file. After createSession, try to save cache to file.

cache_file : the file path to save or load cache.

key_size : the size of key

Note

The API should be called before create session.

Key Depercerate, keeping for future use!

pub fn update_cache_file(&self, session: &mut Session<'_>) -> Result<()>

Update cache file

pub fn wait(&self, session: &Session<'_>)

Wait for all output tensors to be ready after computation

pub fn memory(&self, session: &Session<'_>) -> Result<f32>

Get memory usage of a session in MB

pub fn flops(&self, session: &Session<'_>) -> Result<f32>

Get float operation needed in session in M

pub fn resize_status(&self, session: &Session<'_>) -> Result<ResizeStatus>

Get the resize status

Trait Implementations

impl Debug for Interpreter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for Interpreter

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for Interpreter