Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support for tensorrtllm_backend input template #12569

Open
4 of 5 tasks
Alireza3242 opened this issue Jan 9, 2025 · 1 comment
Open
4 of 5 tasks

support for tensorrtllm_backend input template #12569

Alireza3242 opened this issue Jan 9, 2025 · 1 comment
Labels
💪 enhancement New feature or request

Comments

@Alireza3242
Copy link

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I want to add my tensorrt llm model that is served in tritonserver.
But in tensorrt llm backend project the input template is diferent with the triton that dify supports. you can see it in:
https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/all_models/inflight_batcher_llm/ensemble/config.pbtxt

can you add this template?

2. Additional context or comments

tensorrtllm_backend is officially supported by nvidia. If you add thier template, it will help us.

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
@dosubot dosubot bot added the 💪 enhancement New feature or request label Jan 9, 2025
@crazywoola
Copy link
Member

Please checkout the new version https://github.com/langgenius/dify/releases/tag/1.0.0-beta.1, if you have new model providers or tools that you want to contribute.
#11588

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💪 enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants