Upload folder using huggingface_hub

LICENSE.txt

@@ -0,0 +1,661 @@
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate.  Many developers of free software are heartened and
+encouraged by the resulting cooperation.  However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+  The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community.  It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server.  Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+  An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals.  This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU Affero General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Remote Network Interaction; Use with the GNU General Public License.
+
+  Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software.  This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time.  Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published
+    by the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source.  For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code.  There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<https://www.gnu.org/licenses/>.

SegTracker.py

@@ -0,0 +1,264 @@
+import sys
+sys.path.append("..")
+sys.path.append("./sam")
+from sam.segment_anything import sam_model_registry, SamAutomaticMaskGenerator
+from aot_tracker import get_aot
+import numpy as np
+from tool.segmentor import Segmentor
+from tool.detector import Detector
+from tool.transfer_tools import draw_outline, draw_points
+import cv2
+from seg_track_anything import draw_mask
+
+
+class SegTracker():
+    def __init__(self,segtracker_args, sam_args, aot_args) -> None:
+        """
+         Initialize SAM and AOT.
+        """
+        self.sam = Segmentor(sam_args)
+        self.tracker = get_aot(aot_args)
+        self.detector = Detector(self.sam.device)
+        self.sam_gap = segtracker_args['sam_gap']
+        self.min_area = segtracker_args['min_area']
+        self.max_obj_num = segtracker_args['max_obj_num']
+        self.min_new_obj_iou = segtracker_args['min_new_obj_iou']
+        self.reference_objs_list = []
+        self.object_idx = 1
+        self.curr_idx = 1
+        self.origin_merged_mask = None  # init by segment-everything or update
+        self.first_frame_mask = None
+
+        # debug
+        self.everything_points = []
+        self.everything_labels = []
+        print("SegTracker has been initialized")
+
+    def seg(self,frame):
+        '''
+        Arguments:
+            frame: numpy array (h,w,3)
+        Return:
+            origin_merged_mask: numpy array (h,w)
+        '''
+        frame = frame[:, :, ::-1]
+        anns = self.sam.everything_generator.generate(frame)
+
+        # anns is a list recording all predictions in an image
+        if len(anns) == 0:
+            return
+        # merge all predictions into one mask (h,w)
+        # note that the merged mask may lost some objects due to the overlapping
+        self.origin_merged_mask = np.zeros(anns[0]['segmentation'].shape,dtype=np.uint8)
+        idx = 1
+        for ann in anns:
+            if ann['area'] > self.min_area:
+                m = ann['segmentation']
+                self.origin_merged_mask[m==1] = idx
+                idx += 1
+                self.everything_points.append(ann["point_coords"][0])
+                self.everything_labels.append(1)
+
+        obj_ids = np.unique(self.origin_merged_mask)
+        obj_ids = obj_ids[obj_ids!=0]
+
+        self.object_idx = 1
+        for id in obj_ids:
+            if np.sum(self.origin_merged_mask==id) < self.min_area or self.object_idx > self.max_obj_num:
+                self.origin_merged_mask[self.origin_merged_mask==id] = 0
+            else:
+                self.origin_merged_mask[self.origin_merged_mask==id] = self.object_idx
+                self.object_idx += 1
+
+        self.first_frame_mask = self.origin_merged_mask
+        return self.origin_merged_mask
+
+    def update_origin_merged_mask(self, updated_merged_mask):
+        self.origin_merged_mask = updated_merged_mask
+        # obj_ids = np.unique(updated_merged_mask)
+        # obj_ids = obj_ids[obj_ids!=0]
+        # self.object_idx = int(max(obj_ids)) + 1
+
+    def reset_origin_merged_mask(self, mask, id):
+        self.origin_merged_mask = mask
+        self.curr_idx = id
+
+    def add_reference(self,frame,mask,frame_step=0):
+        '''
+        Add objects in a mask for tracking.
+        Arguments:
+            frame: numpy array (h,w,3)
+            mask: numpy array (h,w)
+        '''
+        self.reference_objs_list.append(np.unique(mask))
+        self.curr_idx = self.get_obj_num() + 1
+        self.tracker.add_reference_frame(frame,mask, self.curr_idx - 1, frame_step)
+
+    def track(self,frame,update_memory=False):
+        '''
+        Track all known objects.
+        Arguments:
+            frame: numpy array (h,w,3)
+        Return:
+            origin_merged_mask: numpy array (h,w)
+        '''
+        pred_mask = self.tracker.track(frame)
+        if update_memory:
+            self.tracker.update_memory(pred_mask)
+        return pred_mask.squeeze(0).squeeze(0).detach().cpu().numpy().astype(np.uint8)
+    
+    def get_tracking_objs(self):
+        objs = set()
+        for ref in self.reference_objs_list:
+            objs.update(set(ref))
+        objs = list(sorted(list(objs)))
+        objs = [i for i in objs if i!=0]
+        return objs
+    
+    def get_obj_num(self):
+        objs = self.get_tracking_objs()
+        if len(objs) == 0: return 0
+        return int(max(objs))
+
+    def find_new_objs(self, track_mask, seg_mask):
+        '''
+        Compare tracked results from AOT with segmented results from SAM. Select objects from background if they are not tracked.
+        Arguments:
+            track_mask: numpy array (h,w)
+            seg_mask: numpy array (h,w)
+        Return:
+            new_obj_mask: numpy array (h,w)
+        '''
+        new_obj_mask = (track_mask==0) * seg_mask
+        new_obj_ids = np.unique(new_obj_mask)
+        new_obj_ids = new_obj_ids[new_obj_ids!=0]
+        # obj_num = self.get_obj_num() + 1
+        obj_num = self.curr_idx
+        for idx in new_obj_ids:
+            new_obj_area = np.sum(new_obj_mask==idx)
+            obj_area = np.sum(seg_mask==idx)
+            if new_obj_area/obj_area < self.min_new_obj_iou or new_obj_area < self.min_area\
+                or obj_num > self.max_obj_num:
+                new_obj_mask[new_obj_mask==idx] = 0
+            else:
+                new_obj_mask[new_obj_mask==idx] = obj_num
+                obj_num += 1
+        return new_obj_mask
+        
+    def restart_tracker(self):
+        self.tracker.restart()
+
+    def seg_acc_bbox(self, origin_frame: np.ndarray, bbox: np.ndarray,):
+        ''''
+        Use bbox-prompt to get mask
+        Parameters:
+            origin_frame: H, W, C
+            bbox: [[x0, y0], [x1, y1]]
+        Return:
+            refined_merged_mask: numpy array (h, w)
+            masked_frame: numpy array (h, w, c)
+        '''
+        # get interactive_mask
+        interactive_mask = self.sam.segment_with_box(origin_frame, bbox)[0]
+        refined_merged_mask = self.add_mask(interactive_mask)
+
+        # draw mask
+        masked_frame = draw_mask(origin_frame.copy(), refined_merged_mask)
+
+        # draw bbox
+        masked_frame = cv2.rectangle(masked_frame, bbox[0], bbox[1], (0, 0, 255))
+
+        return refined_merged_mask, masked_frame
+
+    def seg_acc_click(self, origin_frame: np.ndarray, coords: np.ndarray, modes: np.ndarray, multimask=True):
+        '''
+        Use point-prompt to get mask
+        Parameters:
+            origin_frame: H, W, C
+            coords: nd.array [[x, y]]
+            modes: nd.array [[1]]
+        Return:
+            refined_merged_mask: numpy array (h, w)
+            masked_frame: numpy array (h, w, c)
+        '''
+        # get interactive_mask
+        interactive_mask = self.sam.segment_with_click(origin_frame, coords, modes, multimask)
+
+        refined_merged_mask = self.add_mask(interactive_mask)
+
+        # draw mask
+        masked_frame = draw_mask(origin_frame.copy(), refined_merged_mask)
+
+        # draw points
+        # self.everything_labels = np.array(self.everything_labels).astype(np.int64)
+        # self.everything_points = np.array(self.everything_points).astype(np.int64)
+
+        masked_frame = draw_points(coords, modes, masked_frame)
+
+        # draw outline
+        masked_frame = draw_outline(interactive_mask, masked_frame)
+
+        return refined_merged_mask, masked_frame
+
+    def add_mask(self, interactive_mask: np.ndarray):
+        '''
+        Merge interactive mask with self.origin_merged_mask
+        Parameters:
+            interactive_mask: numpy array (h, w)
+        Return:
+            refined_merged_mask: numpy array (h, w)
+        '''
+        if self.origin_merged_mask is None:
+            self.origin_merged_mask = np.zeros(interactive_mask.shape,dtype=np.uint8)
+
+        refined_merged_mask = self.origin_merged_mask.copy()
+        refined_merged_mask[interactive_mask > 0] = self.curr_idx
+
+        return refined_merged_mask
+    
+    def detect_and_seg(self, origin_frame: np.ndarray, grounding_caption, box_threshold, text_threshold, box_size_threshold=1, reset_image=False):
+        '''
+        Using Grounding-DINO to detect object acc Text-prompts
+        Retrun:
+            refined_merged_mask: numpy array (h, w)
+            annotated_frame: numpy array (h, w, 3)
+        '''
+        # backup id and origin-merged-mask
+        bc_id = self.curr_idx
+        bc_mask = self.origin_merged_mask
+
+        # get annotated_frame and boxes
+        annotated_frame, boxes = self.detector.run_grounding(origin_frame, grounding_caption, box_threshold, text_threshold)
+        for i in range(len(boxes)):
+            bbox = boxes[i]
+            if (bbox[1][0] - bbox[0][0]) * (bbox[1][1] - bbox[0][1]) > annotated_frame.shape[0] * annotated_frame.shape[1] * box_size_threshold:
+                continue
+            interactive_mask = self.sam.segment_with_box(origin_frame, bbox, reset_image)[0]
+            refined_merged_mask = self.add_mask(interactive_mask)
+            self.update_origin_merged_mask(refined_merged_mask)
+            self.curr_idx += 1
+
+        # reset origin_mask
+        self.reset_origin_merged_mask(bc_mask, bc_id)
+
+        return refined_merged_mask, annotated_frame
+
+if __name__ == '__main__':
+    from model_args import segtracker_args,sam_args,aot_args
+
+    Seg_Tracker = SegTracker(segtracker_args, sam_args, aot_args)
+    
+    # ------------------ detect test ----------------------
+    
+    origin_frame = cv2.imread('/data2/cym/Seg_Tra_any/Segment-and-Track-Anything/debug/point.png')
+    origin_frame = cv2.cvtColor(origin_frame, cv2.COLOR_BGR2RGB)
+    grounding_caption = "swan.water"
+    box_threshold = 0.25
+    text_threshold = 0.25
+
+    predicted_mask, annotated_frame = Seg_Tracker.detect_and_seg(origin_frame, grounding_caption, box_threshold, text_threshold)
+    masked_frame = draw_mask(annotated_frame, predicted_mask)
+    origin_frame = cv2.cvtColor(origin_frame, cv2.COLOR_RGB2BGR)
+
+    cv2.imwrite('./debug/masked_frame.png', masked_frame)
+    cv2.imwrite('./debug/x.png', annotated_frame)

aiken

@@ -0,0 +1 @@
+hahahahahahahahaha aiken lox

aot_tracker.py

@@ -0,0 +1,186 @@
+from statistics import mode
+import torch
+import torch.nn.functional as F
+import os
+import sys
+sys.path.append("./aot")
+from aot.networks.engines.aot_engine import AOTEngine,AOTInferEngine
+from aot.networks.engines.deaot_engine import DeAOTEngine,DeAOTInferEngine
+import importlib
+import numpy as np
+from PIL import Image
+from skimage.morphology.binary import binary_dilation
+
+
+np.random.seed(200)
+_palette = ((np.random.random((3*255))*0.7+0.3)*255).astype(np.uint8).tolist()
+_palette = [0,0,0]+_palette
+
+import aot.dataloaders.video_transforms as tr
+from aot.utils.checkpoint import load_network
+from aot.networks.models import build_vos_model
+from aot.networks.engines import build_engine
+from torchvision import transforms
+
+class AOTTracker(object):
+    def __init__(self, cfg, gpu_id=0):
+        self.gpu_id = gpu_id
+        self.model = build_vos_model(cfg.MODEL_VOS, cfg).cuda(gpu_id)
+        self.model, _ = load_network(self.model, cfg.TEST_CKPT_PATH, gpu_id)
+        # self.engine = self.build_tracker_engine(cfg.MODEL_ENGINE,
+        #                            aot_model=self.model,
+        #                            gpu_id=gpu_id,
+        #                            short_term_mem_skip=4,
+        #                            long_term_mem_gap=cfg.TEST_LONG_TERM_MEM_GAP)
+        self.engine = build_engine(cfg.MODEL_ENGINE,
+                                   phase='eval',
+                                   aot_model=self.model,
+                                   gpu_id=gpu_id,
+                                   short_term_mem_skip=1,
+                                   long_term_mem_gap=cfg.TEST_LONG_TERM_MEM_GAP,
+                                   max_len_long_term=cfg.MAX_LEN_LONG_TERM)
+       
+        self.transform = transforms.Compose([
+            tr.MultiRestrictSize(cfg.TEST_MAX_SHORT_EDGE,
+                                 cfg.TEST_MAX_LONG_EDGE, cfg.TEST_FLIP, 
+                                 cfg.TEST_MULTISCALE, cfg.MODEL_ALIGN_CORNERS),
+            tr.MultiToTensor()
+        ])
+
+        self.model.eval()
+
+    @torch.no_grad()
+    def add_reference_frame(self, frame, mask, obj_nums, frame_step, incremental=False):
+        # mask = cv2.resize(mask, frame.shape[:2][::-1], interpolation = cv2.INTER_NEAREST)
+
+        sample = {
+            'current_img': frame,
+            'current_label': mask,
+        }
+    
+        sample = self.transform(sample)
+        frame = sample[0]['current_img'].unsqueeze(0).float().cuda(self.gpu_id)
+        mask = sample[0]['current_label'].unsqueeze(0).float().cuda(self.gpu_id)
+        _mask = F.interpolate(mask,size=frame.shape[-2:],mode='nearest')
+
+        if incremental:
+            self.engine.add_reference_frame_incremental(frame, _mask, obj_nums=obj_nums, frame_step=frame_step)
+        else:
+            self.engine.add_reference_frame(frame, _mask, obj_nums=obj_nums, frame_step=frame_step)
+
+
+
+    @torch.no_grad()
+    def track(self, image):
+        output_height, output_width = image.shape[0], image.shape[1]
+        sample = {'current_img': image}
+        sample = self.transform(sample)
+        image = sample[0]['current_img'].unsqueeze(0).float().cuda(self.gpu_id)
+        self.engine.match_propogate_one_frame(image)
+        pred_logit = self.engine.decode_current_logits((output_height, output_width))
+
+        # pred_prob = torch.softmax(pred_logit, dim=1)
+        pred_label = torch.argmax(pred_logit, dim=1,
+                                    keepdim=True).float()
+
+        return  pred_label
+    
+    @torch.no_grad()
+    def update_memory(self, pred_label):
+        self.engine.update_memory(pred_label)
+    
+    @torch.no_grad()
+    def restart(self):
+        self.engine.restart_engine()
+    
+    @torch.no_grad()
+    def build_tracker_engine(self, name, **kwargs):
+        if name == 'aotengine':
+            return AOTTrackerInferEngine(**kwargs)
+        elif name == 'deaotengine':
+            return DeAOTTrackerInferEngine(**kwargs)
+        else:
+            raise NotImplementedError
+
+
+class AOTTrackerInferEngine(AOTInferEngine):
+    def __init__(self, aot_model, gpu_id=0, long_term_mem_gap=9999, short_term_mem_skip=1, max_aot_obj_num=None):
+        super().__init__(aot_model, gpu_id, long_term_mem_gap, short_term_mem_skip, max_aot_obj_num)
+    def add_reference_frame_incremental(self, img, mask, obj_nums, frame_step=-1):
+        if isinstance(obj_nums, list):
+            obj_nums = obj_nums[0]
+        self.obj_nums = obj_nums
+        aot_num = max(np.ceil(obj_nums / self.max_aot_obj_num), 1)
+        while (aot_num > len(self.aot_engines)):
+            new_engine = AOTEngine(self.AOT, self.gpu_id,
+                                   self.long_term_mem_gap,
+                                   self.short_term_mem_skip)
+            new_engine.eval()
+            self.aot_engines.append(new_engine)
+
+        separated_masks, separated_obj_nums = self.separate_mask(
+            mask, obj_nums)
+        img_embs = None
+        for aot_engine, separated_mask, separated_obj_num in zip(
+                self.aot_engines, separated_masks, separated_obj_nums):
+            if aot_engine.obj_nums is None or aot_engine.obj_nums[0] < separated_obj_num:
+                aot_engine.add_reference_frame(img,
+                                            separated_mask,
+                                            obj_nums=[separated_obj_num],
+                                            frame_step=frame_step,
+                                            img_embs=img_embs)
+            else:
+                aot_engine.update_short_term_memory(separated_mask)
+                
+            if img_embs is None:  # reuse image embeddings
+                img_embs = aot_engine.curr_enc_embs
+
+        self.update_size()
+
+
+
+class DeAOTTrackerInferEngine(DeAOTInferEngine):
+    def __init__(self, aot_model, gpu_id=0, long_term_mem_gap=9999, short_term_mem_skip=1, max_aot_obj_num=None):
+        super().__init__(aot_model, gpu_id, long_term_mem_gap, short_term_mem_skip, max_aot_obj_num)
+    def add_reference_frame_incremental(self, img, mask, obj_nums, frame_step=-1):
+        if isinstance(obj_nums, list):
+            obj_nums = obj_nums[0]
+        self.obj_nums = obj_nums
+        aot_num = max(np.ceil(obj_nums / self.max_aot_obj_num), 1)
+        while (aot_num > len(self.aot_engines)):
+            new_engine = DeAOTEngine(self.AOT, self.gpu_id,
+                                   self.long_term_mem_gap,
+                                   self.short_term_mem_skip)
+            new_engine.eval()
+            self.aot_engines.append(new_engine)
+
+        separated_masks, separated_obj_nums = self.separate_mask(
+            mask, obj_nums)
+        img_embs = None
+        for aot_engine, separated_mask, separated_obj_num in zip(
+                self.aot_engines, separated_masks, separated_obj_nums):
+            if aot_engine.obj_nums is None or aot_engine.obj_nums[0] < separated_obj_num:
+                aot_engine.add_reference_frame(img,
+                                            separated_mask,
+                                            obj_nums=[separated_obj_num],
+                                            frame_step=frame_step,
+                                            img_embs=img_embs)
+            else:
+                aot_engine.update_short_term_memory(separated_mask)
+                
+            if img_embs is None:  # reuse image embeddings
+                img_embs = aot_engine.curr_enc_embs
+
+        self.update_size()
+
+
+def get_aot(args):
+    # build vos engine
+    engine_config = importlib.import_module('configs.' + 'pre_ytb_dav')
+    cfg = engine_config.EngineConfig(args['phase'], args['model'])
+    cfg.TEST_CKPT_PATH = args['model_path']
+    cfg.TEST_LONG_TERM_MEM_GAP = args['long_term_mem_gap']
+    cfg.MAX_LEN_LONG_TERM = args['max_len_long_term']
+    # init AOTTracker
+    tracker = AOTTracker(cfg, args['gpu_id'])
+    return tracker

app.py

@@ -0,0 +1,782 @@
+from PIL.ImageOps import colorize, scale
+import gradio as gr
+import importlib
+import sys
+import os
+
+from matplotlib.pyplot import step
+
+from model_args import segtracker_args,sam_args,aot_args
+from SegTracker import SegTracker
+
+# sys.path.append('.')
+# sys.path.append('..')
+
+import cv2
+from PIL import Image
+from skimage.morphology.binary import binary_dilation
+import argparse
+import torch
+import time
+from seg_track_anything import aot_model2ckpt, tracking_objects_in_video, draw_mask
+import gc
+import numpy as np
+import json
+from tool.transfer_tools import mask2bbox
+
+def clean():
+    return None, None, None, None, None, None, [[], []]
+
+def get_click_prompt(click_stack, point):
+
+    click_stack[0].append(point["coord"])
+    click_stack[1].append(point["mode"]
+    )
+    
+    prompt = {
+        "points_coord":click_stack[0],
+        "points_mode":click_stack[1],
+        "multimask":"True",
+    }
+
+    return prompt
+
+def get_meta_from_video(input_video):
+    if input_video is None:
+        return None, None, None, ""
+
+    print("get meta information of input video")
+    cap = cv2.VideoCapture(input_video)
+    
+    _, first_frame = cap.read()
+    cap.release()
+
+    first_frame = cv2.cvtColor(first_frame, cv2.COLOR_BGR2RGB)
+
+    return first_frame, first_frame, first_frame, ""
+
+def get_meta_from_img_seq(input_img_seq):
+    if input_img_seq is None:
+        return None, None, None, ""
+
+    print("get meta information of img seq")
+    # Create dir
+    file_name = input_img_seq.name.split('/')[-1].split('.')[0]
+    file_path = f'./assets/{file_name}'
+    if os.path.isdir(file_path):
+        os.system(f'rm -r {file_path}')
+    os.makedirs(file_path)
+    # Unzip file
+    os.system(f'unzip {input_img_seq.name} -d ./assets ')
+    
+    imgs_path = sorted([os.path.join(file_path, img_name) for img_name in os.listdir(file_path)])
+    first_frame = imgs_path[0]
+    first_frame = cv2.imread(first_frame)
+    first_frame = cv2.cvtColor(first_frame, cv2.COLOR_BGR2RGB)
+
+    return first_frame, first_frame, first_frame
+
+def SegTracker_add_first_frame(Seg_Tracker, origin_frame, predicted_mask):
+    with torch.cuda.amp.autocast():
+        # Reset the first frame's mask
+        frame_idx = 0
+        Seg_Tracker.restart_tracker()
+        Seg_Tracker.add_reference(origin_frame, predicted_mask, frame_idx)
+        Seg_Tracker.first_frame_mask = predicted_mask
+
+    return Seg_Tracker
+
+def init_SegTracker(aot_model, long_term_mem, max_len_long_term, sam_gap, max_obj_num, points_per_side, origin_frame):
+    
+    if origin_frame is None:
+        return None, origin_frame, [[], []], ""
+
+    # reset aot args
+    aot_args["model"] = aot_model
+    aot_args["model_path"] = aot_model2ckpt[aot_model]
+    aot_args["long_term_mem_gap"] = long_term_mem
+    aot_args["max_len_long_term"] = max_len_long_term
+    # reset sam args
+    segtracker_args["sam_gap"] = sam_gap
+    segtracker_args["max_obj_num"] = max_obj_num
+    sam_args["generator_args"]["points_per_side"] = points_per_side
+    
+    Seg_Tracker = SegTracker(segtracker_args, sam_args, aot_args)
+    Seg_Tracker.restart_tracker()
+
+    return Seg_Tracker, origin_frame, [[], []], ""
+
+def init_SegTracker_Stroke(aot_model, long_term_mem, max_len_long_term, sam_gap, max_obj_num, points_per_side, origin_frame):
+    
+    if origin_frame is None:
+        return None, origin_frame, [[], []], origin_frame
+
+    # reset aot args
+    aot_args["model"] = aot_model
+    aot_args["model_path"] = aot_model2ckpt[aot_model]
+    aot_args["long_term_mem_gap"] = long_term_mem
+    aot_args["max_len_long_term"] = max_len_long_term
+
+    # reset sam args
+    segtracker_args["sam_gap"] = sam_gap
+    segtracker_args["max_obj_num"] = max_obj_num
+    sam_args["generator_args"]["points_per_side"] = points_per_side
+    
+    Seg_Tracker = SegTracker(segtracker_args, sam_args, aot_args)
+    Seg_Tracker.restart_tracker()
+    return Seg_Tracker, origin_frame, [[], []], origin_frame
+
+def undo_click_stack_and_refine_seg(Seg_Tracker, origin_frame, click_stack, aot_model, long_term_mem, max_len_long_term, sam_gap, max_obj_num, points_per_side):
+    
+    if Seg_Tracker is None:
+        return Seg_Tracker, origin_frame, [[], []]
+
+    print("Undo!")
+    if len(click_stack[0]) > 0:
+        click_stack[0] = click_stack[0][: -1]
+        click_stack[1] = click_stack[1][: -1]
+    
+    if len(click_stack[0]) > 0:
+        prompt = {
+            "points_coord":click_stack[0],
+            "points_mode":click_stack[1],
+            "multimask":"True",
+        }
+
+        masked_frame = seg_acc_click(Seg_Tracker, prompt, origin_frame)
+        return Seg_Tracker, masked_frame, click_stack
+    else:
+        return Seg_Tracker, origin_frame, [[], []]
+
+
+def seg_acc_click(Seg_Tracker, prompt, origin_frame):
+    # seg acc to click
+    predicted_mask, masked_frame = Seg_Tracker.seg_acc_click( 
+                                                      origin_frame=origin_frame, 
+                                                      coords=np.array(prompt["points_coord"]),
+                                                      modes=np.array(prompt["points_mode"]),
+                                                      multimask=prompt["multimask"],
+                                                    )
+
+    Seg_Tracker = SegTracker_add_first_frame(Seg_Tracker, origin_frame, predicted_mask)
+
+    return masked_frame
+
+def sam_click(Seg_Tracker, origin_frame, point_mode, click_stack, aot_model, long_term_mem, max_len_long_term, sam_gap, max_obj_num, points_per_side, evt:gr.SelectData):
+    """
+    Args:
+        origin_frame: nd.array
+        click_stack: [[coordinate], [point_mode]]
+    """
+
+    print("Click")
+
+    if point_mode == "Positive":
+        point = {"coord": [evt.index[0], evt.index[1]], "mode": 1}
+    else:
+        # TODO：add everything positive points
+        point = {"coord": [evt.index[0], evt.index[1]], "mode": 0}
+
+    if Seg_Tracker is None:
+        Seg_Tracker, _, _, _ = init_SegTracker(aot_model, long_term_mem, max_len_long_term, sam_gap, max_obj_num, points_per_side, origin_frame)
+
+    # get click prompts for sam to predict mask
+    click_prompt = get_click_prompt(click_stack, point)
+
+    # Refine acc to prompt
+    masked_frame = seg_acc_click(Seg_Tracker, click_prompt, origin_frame)
+
+    return Seg_Tracker, masked_frame, click_stack
+
+def sam_stroke(Seg_Tracker, origin_frame, drawing_board, aot_model, long_term_mem, max_len_long_term, sam_gap, max_obj_num, points_per_side):
+
+    if Seg_Tracker is None:
+        Seg_Tracker, _ , _, _ = init_SegTracker(aot_model, long_term_mem, max_len_long_term, sam_gap, max_obj_num, points_per_side, origin_frame)
+    
+    print("Stroke")
+    mask = drawing_board["mask"]
+    bbox = mask2bbox(mask[:, :, 0])  # bbox: [[x0, y0], [x1, y1]]
+    predicted_mask, masked_frame = Seg_Tracker.seg_acc_bbox(origin_frame, bbox)
+
+    Seg_Tracker = SegTracker_add_first_frame(Seg_Tracker, origin_frame, predicted_mask)
+
+    return Seg_Tracker, masked_frame, origin_frame
+
+def gd_detect(Seg_Tracker, origin_frame, grounding_caption, box_threshold, text_threshold, aot_model, long_term_mem, max_len_long_term, sam_gap, max_obj_num, points_per_side):
+    if Seg_Tracker is None:
+        Seg_Tracker, _ , _, _ = init_SegTracker(aot_model, long_term_mem, max_len_long_term, sam_gap, max_obj_num, points_per_side, origin_frame)
+
+    print("Detect")
+    predicted_mask, annotated_frame= Seg_Tracker.detect_and_seg(origin_frame, grounding_caption, box_threshold, text_threshold)
+
+    Seg_Tracker = SegTracker_add_first_frame(Seg_Tracker, origin_frame, predicted_mask)
+
+
+    masked_frame = draw_mask(annotated_frame, predicted_mask)
+
+    return Seg_Tracker, masked_frame, origin_frame
+
+def segment_everything(Seg_Tracker, aot_model, long_term_mem, max_len_long_term, origin_frame, sam_gap, max_obj_num, points_per_side):
+    
+    if Seg_Tracker is None:
+        Seg_Tracker, _ , _, _ = init_SegTracker(aot_model, long_term_mem, max_len_long_term, sam_gap, max_obj_num, points_per_side, origin_frame)
+
+    print("Everything")
+
+    frame_idx = 0
+
+    with torch.cuda.amp.autocast():
+        pred_mask = Seg_Tracker.seg(origin_frame)
+        torch.cuda.empty_cache()
+        gc.collect()
+        Seg_Tracker.add_reference(origin_frame, pred_mask, frame_idx)
+        Seg_Tracker.first_frame_mask = pred_mask
+
+    masked_frame = draw_mask(origin_frame.copy(), pred_mask)
+
+    return Seg_Tracker, masked_frame
+
+def add_new_object(Seg_Tracker):
+
+    prev_mask = Seg_Tracker.first_frame_mask
+    Seg_Tracker.update_origin_merged_mask(prev_mask)    
+    Seg_Tracker.curr_idx += 1
+
+    print("Ready to add new object!")
+
+    return Seg_Tracker, [[], []]
+
+def tracking_objects(Seg_Tracker, input_video, input_img_seq, fps):
+    print("Start tracking !")
+    return tracking_objects_in_video(Seg_Tracker, input_video, input_img_seq, fps)
+
+def seg_track_app():
+
+    ##########################################################
+    ######################  Front-end ########################
+    ##########################################################
+    app = gr.Blocks()
+
+    with app:
+        gr.Markdown(
+            '''
+            <div style="text-align:center;">
+                <span style="font-size:3em; font-weight:bold;">Segment and Track Anything(SAM-Track)</span>
+            </div>
+            '''
+        )
+
+        click_stack = gr.State([[],[]]) # Storage clicks status
+        origin_frame = gr.State(None)
+        Seg_Tracker = gr.State(None)
+
+        aot_model = gr.State(None)
+        sam_gap = gr.State(None)
+        points_per_side = gr.State(None)
+        max_obj_num = gr.State(None)
+
+        with gr.Row():
+            # video input
+            with gr.Column(scale=0.5):
+
+                tab_video_input = gr.Tab(label="Video type input")
+                with tab_video_input:
+                    input_video = gr.Video(label='Input video').style(height=550)
+                
+                tab_img_seq_input = gr.Tab(label="Image-Seq type input")
+                with tab_img_seq_input:
+                    with gr.Row():
+                        input_img_seq = gr.File(label='Input Image-Seq').style(height=550)
+                        with gr.Column(scale=0.25):
+                            extract_button = gr.Button(value="extract")
+                            fps = gr.Slider(label='fps', minimum=5, maximum=50, value=8, step=1)
+
+                input_first_frame = gr.Image(label='Segment result of first frame',interactive=True).style(height=550)
+
+
+                tab_everything = gr.Tab(label="Everything")
+                with tab_everything:
+                    with gr.Row():
+                        seg_every_first_frame = gr.Button(value="Segment everything for first frame", interactive=True)
+                        point_mode = gr.Radio(
+                            choices=["Positive"],
+                            value="Positive",
+                            label="Point Prompt",
+                            interactive=True)
+
+                        every_undo_but = gr.Button(
+                                    value="Undo",
+                                    interactive=True
+                                    )
+
+                            # every_reset_but = gr.Button(
+                            #             value="Reset",
+                            #             interactive=True
+                            #                     )
+
+                tab_click = gr.Tab(label="Click")
+                with tab_click:
+                    with gr.Row():
+                        point_mode = gr.Radio(
+                                    choices=["Positive",  "Negative"],
+                                    value="Positive",
+                                    label="Point Prompt",
+                                    interactive=True)
+
+                        # args for modify and tracking 
+                        click_undo_but = gr.Button(
+                                    value="Undo",
+                                    interactive=True
+                                    )
+                            # click_reset_but = gr.Button(
+                            #             value="Reset",
+                            #             interactive=True
+                            #                     )
+
+                tab_stroke = gr.Tab(label="Stroke")
+                with tab_stroke:
+                    drawing_board = gr.Image(label='Drawing Board', tool="sketch", brush_radius=10, interactive=True)
+                    with gr.Row():
+                        seg_acc_stroke = gr.Button(value="Segment", interactive=True)
+                        # stroke_reset_but = gr.Button(
+                        #                 value="Reset",
+                        #                 interactive=True
+                        #                         )
+                
+                tab_text = gr.Tab(label="Text")
+                with tab_text:
+                    grounding_caption = gr.Textbox(label="Detection Prompt")
+                    detect_button = gr.Button(value="Detect")
+                    with gr.Accordion("Advanced options", open=False):
+                        with gr.Row():
+                            with gr.Column(scale=0.5):
+                                box_threshold = gr.Slider(
+                                    label="Box Threshold", minimum=0.0, maximum=1.0, value=0.25, step=0.001
+                                )
+                            with gr.Column(scale=0.5):
+                                text_threshold = gr.Slider(
+                                    label="Text Threshold", minimum=0.0, maximum=1.0, value=0.25, step=0.001
+                                )
+
+                with gr.Row():
+                    with gr.Column(scale=0.5):
+                        with gr.Tab(label="SegTracker Args"):
+                            # args for tracking in video do segment-everthing
+                            points_per_side = gr.Slider(
+                                label = "points_per_side",
+                                minimum= 1,
+                                step = 1,
+                                maximum=100,
+                                value=16,
+                                interactive=True
+                            )
+
+                            sam_gap = gr.Slider(
+                                label='sam_gap',
+                                minimum = 1,
+                                step=1,
+                                maximum = 9999,
+                                value=100,
+                                interactive=True,
+                            )
+
+                            max_obj_num = gr.Slider(
+                                label='max_obj_num',
+                                minimum = 50,
+                                step=1,
+                                maximum = 300,
+                                value=255,
+                                interactive=True
+                            )
+                            with gr.Accordion("aot advanced options", open=False):
+                                aot_model = gr.Dropdown(
+                                    label="aot_model",
+                                    choices = [
+                                        "deaotb",
+                                        "deaotl",
+                                        "r50_deaotl"
+                                    ],
+                                    value = "r50_deaotl",
+                                    interactive=True,
+                                )
+                                long_term_mem = gr.Slider(label="long term memory gap", minimum=1, maximum=9999, value=9999, step=1)
+                                max_len_long_term = gr.Slider(label="max len of long term memory", minimum=1, maximum=9999, value=9999, step=1)
+
+                    with gr.Column():
+                        new_object_button = gr.Button(
+                            value="Add new object", 
+                            interactive=True
+                        )
+                        reset_button = gr.Button(
+                            value="Reset",
+                            interactive=True,
+                        )
+                        track_for_video = gr.Button(
+                            value="Start Tracking",
+                                interactive=True,
+                                )
+
+            with gr.Column(scale=0.5):
+                output_video = gr.Video(label='Output video').style(height=550)
+                output_mask = gr.File(label="Predicted masks")
+
+    ##########################################################
+    ######################  back-end #########################
+    ##########################################################
+
+        # listen to the input_video to get the first frame of video
+        input_video.change(
+            fn=get_meta_from_video,
+            inputs=[
+                input_video
+            ],
+            outputs=[
+                input_first_frame, origin_frame, drawing_board, grounding_caption
+            ]
+        )
+
+        # listen to the input_img_seq to get the first frame of video
+        input_img_seq.change(
+            fn=get_meta_from_img_seq,
+            inputs=[
+                input_img_seq
+            ],
+            outputs=[
+                input_first_frame, origin_frame, drawing_board, grounding_caption
+            ]
+        )
+        
+        #-------------- Input compont -------------
+        tab_video_input.select(
+            fn = clean,
+            inputs=[],
+            outputs=[
+                input_video,
+                input_img_seq,
+                Seg_Tracker,
+                input_first_frame,
+                origin_frame,
+                drawing_board,
+                click_stack,
+            ]
+        )
+
+        tab_img_seq_input.select(
+            fn = clean,
+            inputs=[],
+            outputs=[
+                input_video,
+                input_img_seq,
+                Seg_Tracker,
+                input_first_frame,
+                origin_frame,
+                drawing_board,
+                click_stack,
+            ]
+        )
+
+        extract_button.click(
+            fn=get_meta_from_img_seq,
+            inputs=[
+                input_img_seq
+            ],
+            outputs=[
+                input_first_frame, origin_frame, drawing_board
+            ]
+        )
+
+
+        # ------------------- Interactive component -----------------
+
+        # listen to the tab to init SegTracker
+        tab_everything.select(
+            fn=init_SegTracker,
+            inputs=[
+                aot_model,
+                long_term_mem,
+                max_len_long_term, 
+                sam_gap,
+                max_obj_num,
+                points_per_side,
+                origin_frame
+            ],
+            outputs=[
+                Seg_Tracker, input_first_frame, click_stack, grounding_caption
+            ],
+            queue=False,
+            
+        )
+        
+        tab_click.select(
+            fn=init_SegTracker,
+            inputs=[
+                aot_model,
+                long_term_mem,
+                max_len_long_term,
+                sam_gap,
+                max_obj_num,
+                points_per_side,
+                origin_frame
+            ],
+            outputs=[
+                Seg_Tracker, input_first_frame, click_stack, grounding_caption
+            ],
+            queue=False,
+        )
+
+        tab_stroke.select(
+            fn=init_SegTracker_Stroke,
+            inputs=[
+                aot_model,
+                long_term_mem,
+                max_len_long_term,
+                sam_gap,
+                max_obj_num,
+                points_per_side,
+                origin_frame,
+            ],
+            outputs=[
+                Seg_Tracker, input_first_frame, click_stack, drawing_board
+            ],
+            queue=False,
+        )
+
+        tab_text.select(
+            fn=init_SegTracker,
+            inputs=[
+                aot_model,
+                long_term_mem,
+                max_len_long_term,
+                sam_gap,
+                max_obj_num,
+                points_per_side,
+                origin_frame
+            ],
+            outputs=[
+                Seg_Tracker, input_first_frame, click_stack, grounding_caption
+            ],
+            queue=False,
+        )
+
+        # Use SAM to segment everything for the first frame of video
+        seg_every_first_frame.click(
+            fn=segment_everything,
+            inputs=[
+                Seg_Tracker,
+                aot_model,
+                long_term_mem,
+                max_len_long_term,
+                origin_frame,
+                sam_gap,
+                max_obj_num,
+                points_per_side,
+
+            ],
+            outputs=[
+                Seg_Tracker,
+                input_first_frame,
+            ],
+            )
+        
+        # Interactively modify the mask acc click
+        input_first_frame.select(
+            fn=sam_click,
+            inputs=[
+                Seg_Tracker, origin_frame, point_mode, click_stack,
+                aot_model,
+                long_term_mem,
+                max_len_long_term,
+                sam_gap,
+                max_obj_num,
+                points_per_side,
+            ],
+            outputs=[
+                Seg_Tracker, input_first_frame, click_stack
+            ]
+        )
+
+        # Interactively segment acc stroke
+        seg_acc_stroke.click(
+            fn=sam_stroke,
+            inputs=[
+                Seg_Tracker, origin_frame, drawing_board,
+                aot_model,
+                long_term_mem,
+                max_len_long_term,
+                sam_gap,
+                max_obj_num,
+                points_per_side,
+            ],
+            outputs=[
+                Seg_Tracker, input_first_frame, drawing_board
+            ]
+        )
+
+        # Use grounding-dino to detect object
+        detect_button.click(
+            fn=gd_detect, 
+            inputs=[
+                Seg_Tracker, origin_frame, grounding_caption, box_threshold, text_threshold,
+                aot_model, long_term_mem, max_len_long_term, sam_gap, max_obj_num, points_per_side
+                ], 
+            outputs=[
+                Seg_Tracker, input_first_frame
+                ]
+                )
+
+        # Add new object
+        new_object_button.click(
+            fn=add_new_object,
+            inputs=
+            [
+                Seg_Tracker
+            ],
+            outputs=
+            [
+                Seg_Tracker, click_stack
+            ]
+        )
+
+        # Track object in video
+        track_for_video.click(
+            fn=tracking_objects,
+            inputs=[
+                Seg_Tracker,
+                input_video,
+                input_img_seq,
+                fps,
+            ],
+            outputs=[
+                output_video, output_mask
+            ]
+        )
+
+        # ----------------- Reset and Undo ---------------------------
+
+        # Rest 
+        reset_button.click(
+            fn=init_SegTracker,
+            inputs=[
+                aot_model,
+                long_term_mem,
+                max_len_long_term,
+                sam_gap,
+                max_obj_num,
+                points_per_side,
+                origin_frame
+            ],
+            outputs=[
+                Seg_Tracker, input_first_frame, click_stack, grounding_caption
+            ],
+            queue=False,
+            show_progress=False
+        ) 
+
+        # every_reset_but.click(
+        #     fn=init_SegTracker,
+        #     inputs=[
+        #         aot_model,
+        #         sam_gap,
+        #         max_obj_num,
+        #         points_per_side,
+        #         origin_frame
+        #     ],
+        #     outputs=[
+        #         Seg_Tracker, input_first_frame, click_stack, grounding_caption
+        #     ],
+        #     queue=False,
+        #     show_progress=False
+        # ) 
+
+        # click_reset_but.click(
+        #     fn=init_SegTracker,
+        #     inputs=[
+        #         aot_model,
+        #         sam_gap,
+        #         max_obj_num,
+        #         points_per_side,
+        #         origin_frame
+        #     ],
+        #     outputs=[
+        #         Seg_Tracker, input_first_frame, click_stack, grounding_caption
+        #     ],
+        #     queue=False,
+        #     show_progress=False
+        # ) 
+
+        # stroke_reset_but.click(
+        #     fn=init_SegTracker_Stroke,
+        #     inputs=[
+        #         aot_model,
+        #         sam_gap,
+        #         max_obj_num,
+        #         points_per_side,
+        #         origin_frame,
+        #     ],
+        #     outputs=[
+        #         Seg_Tracker, input_first_frame, click_stack, drawing_board
+        #     ],
+        #     queue=False,
+        #     show_progress=False
+        # )
+
+        # Undo click
+        click_undo_but.click(
+            fn = undo_click_stack_and_refine_seg,
+            inputs=[
+                Seg_Tracker, origin_frame, click_stack,
+                aot_model,
+                long_term_mem,
+                max_len_long_term,
+                sam_gap,
+                max_obj_num,
+                points_per_side,
+            ],
+            outputs=[
+               Seg_Tracker, input_first_frame, click_stack
+            ]
+        )
+
+        every_undo_but.click(
+            fn = undo_click_stack_and_refine_seg,
+            inputs=[
+                Seg_Tracker, origin_frame, click_stack,
+                aot_model,
+                long_term_mem,
+                max_len_long_term,
+                sam_gap,
+                max_obj_num,
+                points_per_side,
+            ],
+            outputs=[
+               Seg_Tracker, input_first_frame, click_stack
+            ]
+        )
+        
+        with gr.Tab(label='Video example'):
+            gr.Examples(
+                examples=[
+                    # os.path.join(os.path.dirname(__file__), "assets", "840_iSXIa0hE8Ek.mp4"),
+                    os.path.join(os.path.dirname(__file__), "assets", "blackswan.mp4"),
+                    # os.path.join(os.path.dirname(__file__), "assets", "bear.mp4"),
+                    # os.path.join(os.path.dirname(__file__), "assets", "camel.mp4"),
+                    # os.path.join(os.path.dirname(__file__), "assets", "skate-park.mp4"),
+                    # os.path.join(os.path.dirname(__file__), "assets", "swing.mp4"),
+                    ],
+                inputs=[input_video],
+            )
+        
+        with gr.Tab(label='Image-seq expamle'):
+            gr.Examples(
+                examples=[
+                    os.path.join(os.path.dirname(__file__), "assets", "840_iSXIa0hE8Ek.zip"),
+                ],
+                inputs=[input_img_seq],
+            )
+    
+    app.queue(concurrency_count=1)
+    app.launch(debug=True, enable_queue=True, share=True)
+
+
+if __name__ == "__main__":
+    seg_track_app()

demo.ipynb

@@ -0,0 +1,334 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import cv2\n",
+    "from SegTracker import SegTracker\n",
+    "from model_args import aot_args,sam_args,segtracker_args\n",
+    "from PIL import Image\n",
+    "from aot_tracker import _palette\n",
+    "import numpy as np\n",
+    "import torch\n",
+    "import imageio\n",
+    "import matplotlib.pyplot as plt\n",
+    "from scipy.ndimage import binary_dilation\n",
+    "import gc\n",
+    "def save_prediction(pred_mask,output_dir,file_name):\n",
+    "    save_mask = Image.fromarray(pred_mask.astype(np.uint8))\n",
+    "    save_mask = save_mask.convert(mode='P')\n",
+    "    save_mask.putpalette(_palette)\n",
+    "    save_mask.save(os.path.join(output_dir,file_name))\n",
+    "def colorize_mask(pred_mask):\n",
+    "    save_mask = Image.fromarray(pred_mask.astype(np.uint8))\n",
+    "    save_mask = save_mask.convert(mode='P')\n",
+    "    save_mask.putpalette(_palette)\n",
+    "    save_mask = save_mask.convert(mode='RGB')\n",
+    "    return np.array(save_mask)\n",
+    "def draw_mask(img, mask, alpha=0.5, id_countour=False):\n",
+    "    img_mask = np.zeros_like(img)\n",
+    "    img_mask = img\n",
+    "    if id_countour:\n",
+    "        # very slow ~ 1s per image\n",
+    "        obj_ids = np.unique(mask)\n",
+    "        obj_ids = obj_ids[obj_ids!=0]\n",
+    "\n",
+    "        for id in obj_ids:\n",
+    "            # Overlay color on  binary mask\n",
+    "            if id <= 255:\n",
+    "                color = _palette[id*3:id*3+3]\n",
+    "            else:\n",
+    "                color = [0,0,0]\n",
+    "            foreground = img * (1-alpha) + np.ones_like(img) * alpha * np.array(color)\n",
+    "            binary_mask = (mask == id)\n",
+    "\n",
+    "            # Compose image\n",
+    "            img_mask[binary_mask] = foreground[binary_mask]\n",
+    "\n",
+    "            countours = binary_dilation(binary_mask,iterations=1) ^ binary_mask\n",
+    "            img_mask[countours, :] = 0\n",
+    "    else:\n",
+    "        binary_mask = (mask!=0)\n",
+    "        countours = binary_dilation(binary_mask,iterations=1) ^ binary_mask\n",
+    "        foreground = img*(1-alpha)+colorize_mask(mask)*alpha\n",
+    "        img_mask[binary_mask] = foreground[binary_mask]\n",
+    "        img_mask[countours,:] = 0\n",
+    "        \n",
+    "    return img_mask.astype(img.dtype)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set parameters for input and output"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "video_name = 'cell'\n",
+    "io_args = {\n",
+    "    'input_video': f'./assets/{video_name}.mp4',\n",
+    "    'output_mask_dir': f'./assets/{video_name}_masks', # save pred masks\n",
+    "    'output_video': f'./assets/{video_name}_seg.mp4', # mask+frame vizualization, mp4 or avi, else the same as input video\n",
+    "    'output_gif': f'./assets/{video_name}_seg.gif', # mask visualization\n",
+    "}"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Tuning SAM on the First Frame for Good Initialization"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# choose good parameters in sam_args based on the first frame segmentation result\n",
+    "# other arguments can be modified in model_args.py\n",
+    "# note the object number limit is 255 by default, which requires < 10GB GPU memory with amp\n",
+    "sam_args['generator_args'] = {\n",
+    "        'points_per_side': 30,\n",
+    "        'pred_iou_thresh': 0.8,\n",
+    "        'stability_score_thresh': 0.9,\n",
+    "        'crop_n_layers': 1,\n",
+    "        'crop_n_points_downscale_factor': 2,\n",
+    "        'min_mask_region_area': 200,\n",
+    "    }\n",
+    "cap = cv2.VideoCapture(io_args['input_video'])\n",
+    "frame_idx = 0\n",
+    "segtracker = SegTracker(segtracker_args,sam_args,aot_args)\n",
+    "segtracker.restart_tracker()\n",
+    "with torch.cuda.amp.autocast():\n",
+    "    while cap.isOpened():\n",
+    "        ret, frame = cap.read()\n",
+    "        frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)\n",
+    "        pred_mask = segtracker.seg(frame)\n",
+    "        torch.cuda.empty_cache()\n",
+    "        obj_ids = np.unique(pred_mask)\n",
+    "        obj_ids = obj_ids[obj_ids!=0]\n",
+    "        print(\"processed frame {}, obj_num {}\".format(frame_idx,len(obj_ids)),end='\\n')\n",
+    "        break\n",
+    "    cap.release()\n",
+    "    init_res = draw_mask(frame,pred_mask,id_countour=False)\n",
+    "    plt.figure(figsize=(10,10))\n",
+    "    plt.axis('off')\n",
+    "    plt.imshow(init_res)\n",
+    "    plt.show()\n",
+    "    plt.figure(figsize=(10,10))\n",
+    "    plt.axis('off')\n",
+    "    plt.imshow(colorize_mask(pred_mask))\n",
+    "    plt.show()\n",
+    "\n",
+    "    del segtracker\n",
+    "    torch.cuda.empty_cache()\n",
+    "    gc.collect()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Generate Results for the Whole Video"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# For every sam_gap frames, we use SAM to find new objects and add them for tracking\n",
+    "# larger sam_gap is faster but may not spot new objects in time\n",
+    "segtracker_args = {\n",
+    "    'sam_gap': 5, # the interval to run sam to segment new objects\n",
+    "    'min_area': 200, # minimal mask area to add a new mask as a new object\n",
+    "    'max_obj_num': 255, # maximal object number to track in a video\n",
+    "    'min_new_obj_iou': 0.8, # the area of a new object in the background should > 80% \n",
+    "}\n",
+    "\n",
+    "# source video to segment\n",
+    "cap = cv2.VideoCapture(io_args['input_video'])\n",
+    "fps = cap.get(cv2.CAP_PROP_FPS)\n",
+    "# output masks\n",
+    "output_dir = io_args['output_mask_dir']\n",
+    "if not os.path.exists(output_dir):\n",
+    "    os.makedirs(output_dir)\n",
+    "pred_list = []\n",
+    "masked_pred_list = []\n",
+    "\n",
+    "torch.cuda.empty_cache()\n",
+    "gc.collect()\n",
+    "sam_gap = segtracker_args['sam_gap']\n",
+    "frame_idx = 0\n",
+    "segtracker = SegTracker(segtracker_args,sam_args,aot_args)\n",
+    "segtracker.restart_tracker()\n",
+    "\n",
+    "with torch.cuda.amp.autocast():\n",
+    "    while cap.isOpened():\n",
+    "        ret, frame = cap.read()\n",
+    "        if not ret:\n",
+    "            break\n",
+    "        frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)\n",
+    "        if frame_idx == 0:\n",
+    "            pred_mask = segtracker.seg(frame)\n",
+    "            torch.cuda.empty_cache()\n",
+    "            gc.collect()\n",
+    "            segtracker.add_reference(frame, pred_mask)\n",
+    "        elif (frame_idx % sam_gap) == 0:\n",
+    "            seg_mask = segtracker.seg(frame)\n",
+    "            torch.cuda.empty_cache()\n",
+    "            gc.collect()\n",
+    "            track_mask = segtracker.track(frame)\n",
+    "            # find new objects, and update tracker with new objects\n",
+    "            new_obj_mask = segtracker.find_new_objs(track_mask,seg_mask)\n",
+    "            save_prediction(new_obj_mask,output_dir,str(frame_idx)+'_new.png')\n",
+    "            pred_mask = track_mask + new_obj_mask\n",
+    "            # segtracker.restart_tracker()\n",
+    "            segtracker.add_reference(frame, pred_mask)\n",
+    "        else:\n",
+    "            pred_mask = segtracker.track(frame,update_memory=True)\n",
+    "        torch.cuda.empty_cache()\n",
+    "        gc.collect()\n",
+    "        save_prediction(pred_mask,output_dir,str(frame_idx)+'.png')\n",
+    "        # masked_frame = draw_mask(frame,pred_mask)\n",
+    "        # masked_pred_list.append(masked_frame)\n",
+    "        # plt.imshow(masked_frame)\n",
+    "        # plt.show() \n",
+    "        \n",
+    "        pred_list.append(pred_mask)\n",
+    "        \n",
+    "        \n",
+    "        print(\"processed frame {}, obj_num {}\".format(frame_idx,segtracker.get_obj_num()),end='\\r')\n",
+    "        frame_idx += 1\n",
+    "    cap.release()\n",
+    "    print('\\nfinished')"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Save results for visualization"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# draw pred mask on frame and save as a video\n",
+    "cap = cv2.VideoCapture(io_args['input_video'])\n",
+    "fps = cap.get(cv2.CAP_PROP_FPS)\n",
+    "width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))\n",
+    "height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))\n",
+    "num_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n",
+    "\n",
+    "if io_args['input_video'][-3:]=='mp4':\n",
+    "    fourcc =  cv2.VideoWriter_fourcc(*\"mp4v\")\n",
+    "elif io_args['input_video'][-3:] == 'avi':\n",
+    "    fourcc =  cv2.VideoWriter_fourcc(*\"MJPG\")\n",
+    "    # fourcc = cv2.VideoWriter_fourcc(*\"XVID\")\n",
+    "else:\n",
+    "    fourcc = int(cap.get(cv2.CAP_PROP_FOURCC))\n",
+    "out = cv2.VideoWriter(io_args['output_video'], fourcc, fps, (width, height))\n",
+    "\n",
+    "frame_idx = 0\n",
+    "while cap.isOpened():\n",
+    "    ret, frame = cap.read()\n",
+    "    if not ret:\n",
+    "        break\n",
+    "    frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)\n",
+    "    pred_mask = pred_list[frame_idx]\n",
+    "    masked_frame = draw_mask(frame,pred_mask)\n",
+    "    # masked_frame = masked_pred_list[frame_idx]\n",
+    "    masked_frame = cv2.cvtColor(masked_frame,cv2.COLOR_RGB2BGR)\n",
+    "    out.write(masked_frame)\n",
+    "    print('frame {} writed'.format(frame_idx),end='\\r')\n",
+    "    frame_idx += 1\n",
+    "out.release()\n",
+    "cap.release()\n",
+    "print(\"\\n{} saved\".format(io_args['output_video']))\n",
+    "print('\\nfinished')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# save colorized masks as a gif\n",
+    "imageio.mimsave(io_args['output_gif'],pred_list,fps=fps)\n",
+    "print(\"{} saved\".format(io_args['output_gif']))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "301"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# manually release memory (after cuda out of memory)\n",
+    "del segtracker\n",
+    "torch.cuda.empty_cache()\n",
+    "gc.collect()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.8.5 64-bit ('ldm': conda)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.5"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "536611da043600e50719c9460971b5220bad26cd4a87e5994bfd4c9e9e5e7fb0"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

demo_instseg.ipynb

@@ -0,0 +1,353 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import cv2\n",
+    "from SegTracker import SegTracker\n",
+    "from model_args import aot_args,sam_args,segtracker_args\n",
+    "from PIL import Image\n",
+    "from aot_tracker import _palette\n",
+    "import numpy as np\n",
+    "import torch\n",
+    "import imageio\n",
+    "import matplotlib.pyplot as plt\n",
+    "from scipy.ndimage import binary_dilation\n",
+    "import gc\n",
+    "def save_prediction(pred_mask,output_dir,file_name):\n",
+    "    save_mask = Image.fromarray(pred_mask.astype(np.uint8))\n",
+    "    save_mask = save_mask.convert(mode='P')\n",
+    "    save_mask.putpalette(_palette)\n",
+    "    save_mask.save(os.path.join(output_dir,file_name))\n",
+    "def colorize_mask(pred_mask):\n",
+    "    save_mask = Image.fromarray(pred_mask.astype(np.uint8))\n",
+    "    save_mask = save_mask.convert(mode='P')\n",
+    "    save_mask.putpalette(_palette)\n",
+    "    save_mask = save_mask.convert(mode='RGB')\n",
+    "    return np.array(save_mask)\n",
+    "def draw_mask(img, mask, alpha=0.7, id_countour=False):\n",
+    "    img_mask = np.zeros_like(img)\n",
+    "    img_mask = img\n",
+    "    if id_countour:\n",
+    "        # very slow ~ 1s per image\n",
+    "        obj_ids = np.unique(mask)\n",
+    "        obj_ids = obj_ids[obj_ids!=0]\n",
+    "\n",
+    "        for id in obj_ids:\n",
+    "            # Overlay color on  binary mask\n",
+    "            if id <= 255:\n",
+    "                color = _palette[id*3:id*3+3]\n",
+    "            else:\n",
+    "                color = [0,0,0]\n",
+    "            foreground = img * (1-alpha) + np.ones_like(img) * alpha * np.array(color)\n",
+    "            binary_mask = (mask == id)\n",
+    "\n",
+    "            # Compose image\n",
+    "            img_mask[binary_mask] = foreground[binary_mask]\n",
+    "\n",
+    "            countours = binary_dilation(binary_mask,iterations=1) ^ binary_mask\n",
+    "            img_mask[countours, :] = 0\n",
+    "    else:\n",
+    "        binary_mask = (mask!=0)\n",
+    "        countours = binary_dilation(binary_mask,iterations=1) ^ binary_mask\n",
+    "        foreground = img*(1-alpha)+colorize_mask(mask)*alpha\n",
+    "        img_mask[binary_mask] = foreground[binary_mask]\n",
+    "        img_mask[countours,:] = 0\n",
+    "        \n",
+    "    return img_mask.astype(img.dtype)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set parameters for input and output"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "video_name = 'cars'\n",
+    "io_args = {\n",
+    "    'input_video': f'./assets/{video_name}.mp4',\n",
+    "    'output_mask_dir': f'./assets/{video_name}_masks', # save pred masks\n",
+    "    'output_video': f'./assets/{video_name}_seg.mp4', # mask+frame vizualization, mp4 or avi, else the same as input video\n",
+    "    'output_gif': f'./assets/{video_name}_seg.gif', # mask visualization\n",
+    "}"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Tuning Grounding-DINO and SAM on the First Frame for Good Initialization"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# choose good parameters in sam_args based on the first frame segmentation result\n",
+    "# other arguments can be modified in model_args.py\n",
+    "# note the object number limit is 255 by default, which requires < 10GB GPU memory with amp\n",
+    "sam_args['generator_args'] = {\n",
+    "        'points_per_side': 30,\n",
+    "        'pred_iou_thresh': 0.8,\n",
+    "        'stability_score_thresh': 0.9,\n",
+    "        'crop_n_layers': 1,\n",
+    "        'crop_n_points_downscale_factor': 2,\n",
+    "        'min_mask_region_area': 200,\n",
+    "    }\n",
+    "\n",
+    "# Set Text args\n",
+    "'''\n",
+    "parameter:\n",
+    "    grounding_caption: Text prompt to detect objects in key-frames\n",
+    "    box_threshold: threshold for box \n",
+    "    text_threshold: threshold for label(text)\n",
+    "    box_size_threshold: If the size ratio between the box and the frame is larger than the box_size_threshold, the box will be ignored. This is used to filter out large boxes.\n",
+    "    reset_image: reset the image embeddings for SAM\n",
+    "'''\n",
+    "grounding_caption = \"car.suv\"\n",
+    "box_threshold, text_threshold, box_size_threshold, reset_image = 0.35, 0.5, 0.5, True\n",
+    "\n",
+    "cap = cv2.VideoCapture(io_args['input_video'])\n",
+    "frame_idx = 0\n",
+    "segtracker = SegTracker(segtracker_args,sam_args,aot_args)\n",
+    "segtracker.restart_tracker()\n",
+    "with torch.cuda.amp.autocast():\n",
+    "    while cap.isOpened():\n",
+    "        ret, frame = cap.read()\n",
+    "        frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)\n",
+    "        pred_mask, annotated_frame = segtracker.detect_and_seg(frame, grounding_caption, box_threshold, text_threshold, box_size_threshold)\n",
+    "        torch.cuda.empty_cache()\n",
+    "        obj_ids = np.unique(pred_mask)\n",
+    "        obj_ids = obj_ids[obj_ids!=0]\n",
+    "        print(\"processed frame {}, obj_num {}\".format(frame_idx,len(obj_ids)),end='\\n')\n",
+    "        break\n",
+    "    cap.release()\n",
+    "    init_res = draw_mask(annotated_frame, pred_mask,id_countour=False)\n",
+    "    plt.figure(figsize=(10,10))\n",
+    "    plt.axis('off')\n",
+    "    plt.imshow(init_res)\n",
+    "    plt.show()\n",
+    "    plt.figure(figsize=(10,10))\n",
+    "    plt.axis('off')\n",
+    "    plt.imshow(colorize_mask(pred_mask))\n",
+    "    plt.show()\n",
+    "\n",
+    "    del segtracker\n",
+    "    torch.cuda.empty_cache()\n",
+    "    gc.collect()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Generate Results for the Whole Video"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# For every sam_gap frames, we use SAM to find new objects and add them for tracking\n",
+    "# larger sam_gap is faster but may not spot new objects in time\n",
+    "segtracker_args = {\n",
+    "    'sam_gap': 49, # the interval to run sam to segment new objects\n",
+    "    'min_area': 200, # minimal mask area to add a new mask as a new object\n",
+    "    'max_obj_num': 255, # maximal object number to track in a video\n",
+    "    'min_new_obj_iou': 0.8, # the area of a new object in the background should > 80% \n",
+    "}\n",
+    "\n",
+    "# source video to segment\n",
+    "cap = cv2.VideoCapture(io_args['input_video'])\n",
+    "fps = cap.get(cv2.CAP_PROP_FPS)\n",
+    "# output masks\n",
+    "output_dir = io_args['output_mask_dir']\n",
+    "if not os.path.exists(output_dir):\n",
+    "    os.makedirs(output_dir)\n",
+    "pred_list = []\n",
+    "masked_pred_list = []\n",
+    "\n",
+    "torch.cuda.empty_cache()\n",
+    "gc.collect()\n",
+    "sam_gap = segtracker_args['sam_gap']\n",
+    "frame_idx = 0\n",
+    "segtracker = SegTracker(segtracker_args, sam_args, aot_args)\n",
+    "segtracker.restart_tracker()\n",
+    "\n",
+    "with torch.cuda.amp.autocast():\n",
+    "    while cap.isOpened():\n",
+    "        ret, frame = cap.read()\n",
+    "        if not ret:\n",
+    "            break\n",
+    "        frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)\n",
+    "        if frame_idx == 0:\n",
+    "            pred_mask, _ = segtracker.detect_and_seg(frame, grounding_caption, box_threshold, text_threshold, box_size_threshold, reset_image)\n",
+    "            # pred_mask = cv2.imread('./debug/first_frame_mask.png', 0)\n",
+    "            torch.cuda.empty_cache()\n",
+    "            gc.collect()\n",
+    "            segtracker.add_reference(frame, pred_mask)\n",
+    "        elif (frame_idx % sam_gap) == 0:\n",
+    "            seg_mask, _ = segtracker.detect_and_seg(frame, grounding_caption, box_threshold, text_threshold, box_size_threshold, reset_image)\n",
+    "            save_prediction(seg_mask, './debug/seg_result', str(frame_idx)+'.png')\n",
+    "            torch.cuda.empty_cache()\n",
+    "            gc.collect()\n",
+    "            track_mask = segtracker.track(frame)\n",
+    "            save_prediction(track_mask, './debug/aot_result', str(frame_idx)+'.png')\n",
+    "            # find new objects, and update tracker with new objects\n",
+    "            new_obj_mask = segtracker.find_new_objs(track_mask, seg_mask)\n",
+    "            if np.sum(new_obj_mask > 0) >  frame.shape[0] * frame.shape[1] * 0.4:\n",
+    "                new_obj_mask = np.zeros_like(new_obj_mask)\n",
+    "            save_prediction(new_obj_mask,output_dir,str(frame_idx)+'_new.png')\n",
+    "            pred_mask = track_mask + new_obj_mask\n",
+    "            # segtracker.restart_tracker()\n",
+    "            segtracker.add_reference(frame, pred_mask)\n",
+    "        else:\n",
+    "            pred_mask = segtracker.track(frame,update_memory=True)\n",
+    "        torch.cuda.empty_cache()\n",
+    "        gc.collect()\n",
+    "        \n",
+    "        save_prediction(pred_mask,output_dir,str(frame_idx)+'.png')\n",
+    "        # masked_frame = draw_mask(frame,pred_mask)\n",
+    "        # masked_pred_list.append(masked_frame)\n",
+    "        # plt.imshow(masked_frame)\n",
+    "        # plt.show() \n",
+    "        \n",
+    "        pred_list.append(pred_mask)\n",
+    "        \n",
+    "        \n",
+    "        print(\"processed frame {}, obj_num {}\".format(frame_idx,segtracker.get_obj_num()),end='\\r')\n",
+    "        frame_idx += 1\n",
+    "    cap.release()\n",
+    "    print('\\nfinished')"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Save results for visualization"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# draw pred mask on frame and save as a video\n",
+    "cap = cv2.VideoCapture(io_args['input_video'])\n",
+    "fps = cap.get(cv2.CAP_PROP_FPS)\n",
+    "width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))\n",
+    "height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))\n",
+    "num_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))\n",
+    "\n",
+    "if io_args['input_video'][-3:]=='mp4':\n",
+    "    fourcc =  cv2.VideoWriter_fourcc(*\"mp4v\")\n",
+    "elif io_args['input_video'][-3:] == 'avi':\n",
+    "    fourcc =  cv2.VideoWriter_fourcc(*\"MJPG\")\n",
+    "    # fourcc = cv2.VideoWriter_fourcc(*\"XVID\")\n",
+    "else:\n",
+    "    fourcc = int(cap.get(cv2.CAP_PROP_FOURCC))\n",
+    "out = cv2.VideoWriter(io_args['output_video'], fourcc, fps, (width, height))\n",
+    "\n",
+    "frame_idx = 0\n",
+    "while cap.isOpened():\n",
+    "    ret, frame = cap.read()\n",
+    "    if not ret:\n",
+    "        break\n",
+    "    frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)\n",
+    "    pred_mask = pred_list[frame_idx]\n",
+    "    masked_frame = draw_mask(frame,pred_mask)\n",
+    "    # masked_frame = masked_pred_list[frame_idx]\n",
+    "    masked_frame = cv2.cvtColor(masked_frame,cv2.COLOR_RGB2BGR)\n",
+    "    out.write(masked_frame)\n",
+    "    print('frame {} writed'.format(frame_idx),end='\\r')\n",
+    "    frame_idx += 1\n",
+    "out.release()\n",
+    "cap.release()\n",
+    "print(\"\\n{} saved\".format(io_args['output_video']))\n",
+    "print('\\nfinished')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# save colorized masks as a gif\n",
+    "imageio.mimsave(io_args['output_gif'],pred_list,fps=fps)\n",
+    "print(\"{} saved\".format(io_args['output_gif']))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "21"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# manually release memory (after cuda out of memory)\n",
+    "del segtracker\n",
+    "torch.cuda.empty_cache()\n",
+    "gc.collect()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.8.5 64-bit ('ldm': conda)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.5-final"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "536611da043600e50719c9460971b5220bad26cd4a87e5994bfd4c9e9e5e7fb0"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

img2vid.py

@@ -0,0 +1,26 @@
+import cv2
+import os
+
+# set the directory containing the images
+img_dir = './assets/840_iSXIa0hE8Ek'
+
+# set the output video file name and codec
+out_file = './assets/840_iSXIa0hE8Ek.mp4'
+fourcc = cv2.VideoWriter_fourcc(*'mp4v')
+
+# get the dimensions of the first image
+img_path = os.path.join(img_dir, os.listdir(img_dir)[0])
+img = cv2.imread(img_path)
+height, width, channels = img.shape
+
+# create the VideoWriter object
+out = cv2.VideoWriter(out_file, fourcc, 10, (width, height))
+
+# loop through the images and write them to the video
+for img_name in sorted(os.listdir(img_dir)):
+    img_path = os.path.join(img_dir, img_name)
+    img = cv2.imread(img_path)
+    out.write(img)
+
+# release the VideoWriter object and close the video file
+out.release()

licenses.md

@@ -0,0 +1,660 @@
+## <center> [SAM](https://github.com/facebookresearch/segment-anything/blob/main/LICENSE) </center>
+<div align=center>
+
+Apache License
+
+Version 2.0, January 2004
+
+ http://www.apache.org/licenses/           
+</div>
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+   To apply the Apache License to your work, attach the following
+   boilerplate notice, with the fields enclosed by brackets "[]"
+   replaced with your own identifying information. (Don't include
+   the brackets!)  The text should be enclosed in the appropriate
+   comment syntax for the file format. We also recommend that a
+   file or class name and description of purpose be included on the
+   same "printed page" as the copyright notice for easier
+   identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+
+
+## <center> [AOT](https://github.com/yoxu515/aot-benchmark/blob/main/LICENSE) </center>
+<div align=center>
+BSD 3-Clause License
+
+Copyright (c) 2020, z-x-yang
+
+Copyright (c) All rights reserved.
+</div>
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+## <center> [Gradio](https://github.com/gradio-app/gradio/blob/main/LICENSE) </center>
+
+<div align=center>
+Apache License
+
+Version 2.0, January 2004
+
+http://www.apache.org/licenses/
+</div>
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+   To apply the Apache License to your work, attach the following
+   boilerplate notice, with the fields enclosed by brackets "[]"
+   replaced with your own identifying information. (Don't include
+   the brackets!)  The text should be enclosed in the appropriate
+   comment syntax for the file format. We also recommend that a
+   file or class name and description of purpose be included on the
+   same "printed page" as the copyright notice for easier
+   identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+
+   ## <center> [GroundingDINO](https://github.com/yamy-cheng/GroundingDINO/blob/main/LICENSE) </center>
+
+<div align="center">
+ Apache License
+
+Version 2.0, January 2004
+
+http://www.apache.org/licenses/
+
+</div>
+
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+   To apply the Apache License to your work, attach the following
+   boilerplate notice, with the fields enclosed by brackets "[]"
+   replaced with your own identifying information. (Don't include
+   the brackets!)  The text should be enclosed in the appropriate
+   comment syntax for the file format. We also recommend that a
+   file or class name and description of purpose be included on the
+   same "printed page" as the copyright notice for easier
+   identification within third-party archives.
+
+   Copyright 2020 - present, Facebook, Inc
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

model_args.py

@@ -0,0 +1,28 @@
+# Explanation of generator_args is in sam/segment_anything/automatic_mask_generator.py: SamAutomaticMaskGenerator
+sam_args = {
+    'sam_checkpoint': "ckpt/sam_vit_b_01ec64.pth",
+    'model_type': "vit_b",
+    'generator_args':{
+        'points_per_side': 16,
+        'pred_iou_thresh': 0.8,
+        'stability_score_thresh': 0.9,
+        'crop_n_layers': 1,
+        'crop_n_points_downscale_factor': 2,
+        'min_mask_region_area': 200,
+    },
+    'gpu_id': 0,
+}
+aot_args = {
+    'phase': 'PRE_YTB_DAV',
+    'model': 'r50_deaotl',
+    'model_path': 'ckpt/R50_DeAOTL_PRE_YTB_DAV.pth',
+    'long_term_mem_gap': 9999,
+    'max_len_long_term': 9999,
+    'gpu_id': 0,
+}
+segtracker_args = {
+    'sam_gap': 10, # the interval to run sam to segment new objects
+    'min_area': 200, # minimal mask area to add a new mask as a new object
+    'max_obj_num': 255, # maximal object number to track in a video
+    'min_new_obj_iou': 0.8, # the background area ratio of a new object should > 80% 
+}

seg_track_anything.py

@@ -0,0 +1,300 @@
+import os
+import cv2
+from model_args import segtracker_args,sam_args,aot_args
+from PIL import Image
+from aot_tracker import _palette
+import numpy as np
+import torch
+import gc
+import imageio
+from scipy.ndimage import binary_dilation
+
+def save_prediction(pred_mask,output_dir,file_name):
+    save_mask = Image.fromarray(pred_mask.astype(np.uint8))
+    save_mask = save_mask.convert(mode='P')
+    save_mask.putpalette(_palette)
+    save_mask.save(os.path.join(output_dir,file_name))
+
+def colorize_mask(pred_mask):
+    save_mask = Image.fromarray(pred_mask.astype(np.uint8))
+    save_mask = save_mask.convert(mode='P')
+    save_mask.putpalette(_palette)
+    save_mask = save_mask.convert(mode='RGB')
+    return np.array(save_mask)
+
+def draw_mask(img, mask, alpha=0.5, id_countour=False):
+    img_mask = np.zeros_like(img)
+    img_mask = img
+    if id_countour:
+        # very slow ~ 1s per image
+        obj_ids = np.unique(mask)
+        obj_ids = obj_ids[obj_ids!=0]
+
+        for id in obj_ids:
+            # Overlay color on  binary mask
+            if id <= 255:
+                color = _palette[id*3:id*3+3]
+            else:
+                color = [0,0,0]
+            foreground = img * (1-alpha) + np.ones_like(img) * alpha * np.array(color)
+            binary_mask = (mask == id)
+
+            # Compose image
+            img_mask[binary_mask] = foreground[binary_mask]
+
+            countours = binary_dilation(binary_mask,iterations=1) ^ binary_mask
+            img_mask[countours, :] = 0
+    else:
+        binary_mask = (mask!=0)
+        countours = binary_dilation(binary_mask,iterations=1) ^ binary_mask
+        foreground = img*(1-alpha)+colorize_mask(mask)*alpha
+        img_mask[binary_mask] = foreground[binary_mask]
+        img_mask[countours,:] = 0
+        
+    return img_mask.astype(img.dtype)
+
+def create_dir(dir_path):
+    if os.path.isdir(dir_path):
+        os.system(f"rm -r {dir_path}")
+    
+    os.makedirs(dir_path)
+
+aot_model2ckpt = {
+    "deaotb": "./ckpt/DeAOTB_PRE_YTB_DAV.pth",
+    "deaotl": "./ckpt/DeAOTL_PRE_YTB_DAV",
+    "r50_deaotl": "./ckpt/R50_DeAOTL_PRE_YTB_DAV.pth",
+}
+
+
+def tracking_objects_in_video(SegTracker, input_video, input_img_seq, fps):
+    
+    if input_video is not None:
+        video_name = os.path.basename(input_video).split('.')[0]
+    elif input_img_seq is not None:
+        file_name = input_img_seq.name.split('/')[-1].split('.')[0]
+        file_path = f'./assets/{file_name}'
+        imgs_path = sorted([os.path.join(file_path, img_name) for img_name in os.listdir(file_path)])
+        video_name = file_name
+    else:
+        return None, None
+
+    # create dir to save result 
+    tracking_result_dir = f'{os.path.join(os.path.dirname(__file__), "tracking_results", f"{video_name}")}'
+    create_dir(tracking_result_dir)
+    
+    io_args = {
+        'tracking_result_dir': tracking_result_dir,
+        'output_mask_dir': f'{tracking_result_dir}/{video_name}_masks',
+        'output_masked_frame_dir': f'{tracking_result_dir}/{video_name}_masked_frames',
+        'output_video': f'{tracking_result_dir}/{video_name}_seg.mp4', # keep same format as input video
+        'output_gif': f'{tracking_result_dir}/{video_name}_seg.gif',
+    }
+
+    if input_video is not None:
+        return video_type_input_tracking(SegTracker, input_video, io_args, video_name)
+    elif input_img_seq is not None:
+        return img_seq_type_input_tracking(SegTracker, io_args, video_name, imgs_path, fps)
+
+
+def video_type_input_tracking(SegTracker, input_video, io_args, video_name):
+
+    # source video to segment
+    cap = cv2.VideoCapture(input_video)
+    fps = cap.get(cv2.CAP_PROP_FPS)
+    
+    # create dir to save predicted mask and masked frame
+    output_mask_dir = io_args['output_mask_dir']
+    create_dir(io_args['output_mask_dir'])
+    create_dir(io_args['output_masked_frame_dir'])
+
+    pred_list = []
+    masked_pred_list = []
+
+    torch.cuda.empty_cache()
+    gc.collect()
+    sam_gap = SegTracker.sam_gap
+    frame_idx = 0
+
+    with torch.cuda.amp.autocast():
+        while cap.isOpened():
+            ret, frame = cap.read()
+            if not ret:
+                break
+            frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
+            
+            if frame_idx == 0:
+                pred_mask = SegTracker.first_frame_mask
+                torch.cuda.empty_cache()
+                gc.collect()
+            elif (frame_idx % sam_gap) == 0:
+                seg_mask = SegTracker.seg(frame)
+                torch.cuda.empty_cache()
+                gc.collect()
+                track_mask = SegTracker.track(frame)
+                # find new objects, and update tracker with new objects
+                new_obj_mask = SegTracker.find_new_objs(track_mask,seg_mask)
+                save_prediction(new_obj_mask, output_mask_dir, str(frame_idx).zfill(5) + '_new.png')
+                pred_mask = track_mask + new_obj_mask
+                # segtracker.restart_tracker()
+                SegTracker.add_reference(frame, pred_mask)
+            else:
+                pred_mask = SegTracker.track(frame,update_memory=True)
+            torch.cuda.empty_cache()
+            gc.collect()
+            
+            save_prediction(pred_mask, output_mask_dir, str(frame_idx).zfill(5) + '.png')
+            pred_list.append(pred_mask)
+
+            print("processed frame {}, obj_num {}".format(frame_idx, SegTracker.get_obj_num()),end='\r')
+            frame_idx += 1
+        cap.release()
+        print('\nfinished')
+    
+    ##################
+    # Visualization
+    ##################
+
+    # draw pred mask on frame and save as a video
+    cap = cv2.VideoCapture(input_video)
+    fps = cap.get(cv2.CAP_PROP_FPS)
+    width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
+    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
+    num_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
+
+    fourcc =  cv2.VideoWriter_fourcc(*"mp4v")
+    # if input_video[-3:]=='mp4':
+    #     fourcc =  cv2.VideoWriter_fourcc(*"mp4v")
+    # elif input_video[-3:] == 'avi':
+    #     fourcc =  cv2.VideoWriter_fourcc(*"MJPG")
+    #     # fourcc = cv2.VideoWriter_fourcc(*"XVID")
+    # else:
+    #     fourcc = int(cap.get(cv2.CAP_PROP_FOURCC))
+    out = cv2.VideoWriter(io_args['output_video'], fourcc, fps, (width, height))
+
+    frame_idx = 0
+    while cap.isOpened():
+        ret, frame = cap.read()
+        if not ret:
+            break
+
+        frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
+        pred_mask = pred_list[frame_idx]
+        masked_frame = draw_mask(frame, pred_mask)
+        cv2.imwrite(f"{io_args['output_masked_frame_dir']}/{str(frame_idx).zfill(5)}.png", masked_frame[:, :, ::-1])
+
+        masked_pred_list.append(masked_frame)
+        masked_frame = cv2.cvtColor(masked_frame,cv2.COLOR_RGB2BGR)
+        out.write(masked_frame)
+        print('frame {} writed'.format(frame_idx),end='\r')
+        frame_idx += 1
+    out.release()
+    cap.release()
+    print("\n{} saved".format(io_args['output_video']))
+    print('\nfinished')
+
+    # save colorized masks as a gif
+    imageio.mimsave(io_args['output_gif'], masked_pred_list, fps=fps)
+    print("{} saved".format(io_args['output_gif']))
+
+    # zip predicted mask
+    os.system(f"zip -r {io_args['tracking_result_dir']}/{video_name}_pred_mask.zip {io_args['output_mask_dir']}")
+
+    # manually release memory (after cuda out of memory)
+    del SegTracker
+    torch.cuda.empty_cache()
+    gc.collect()
+
+    return io_args['output_video'], f"{io_args['tracking_result_dir']}/{video_name}_pred_mask.zip"
+
+
+def img_seq_type_input_tracking(SegTracker, io_args, video_name, imgs_path, fps):
+
+    # create dir to save predicted mask and masked frame
+    output_mask_dir = io_args['output_mask_dir']
+    create_dir(io_args['output_mask_dir'])
+    create_dir(io_args['output_masked_frame_dir'])
+
+    pred_list = []
+    masked_pred_list = []
+
+    torch.cuda.empty_cache()
+    gc.collect()
+    sam_gap = SegTracker.sam_gap
+    frame_idx = 0
+
+    with torch.cuda.amp.autocast():
+        for img_path in imgs_path:
+            frame_name = os.path.basename(img_path).split('.')[0]
+            frame = cv2.imread(img_path)
+            frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
+            
+            if frame_idx == 0:
+                pred_mask = SegTracker.first_frame_mask
+                torch.cuda.empty_cache()
+                gc.collect()
+            elif (frame_idx % sam_gap) == 0:
+                seg_mask = SegTracker.seg(frame)
+                torch.cuda.empty_cache()
+                gc.collect()
+                track_mask = SegTracker.track(frame)
+                # find new objects, and update tracker with new objects
+                new_obj_mask = SegTracker.find_new_objs(track_mask,seg_mask)
+                save_prediction(new_obj_mask, output_mask_dir, f'{frame_name}_new.png')
+                pred_mask = track_mask + new_obj_mask
+                # segtracker.restart_tracker()
+                SegTracker.add_reference(frame, pred_mask)
+            else:
+                pred_mask = SegTracker.track(frame,update_memory=True)
+            torch.cuda.empty_cache()
+            gc.collect()
+            
+            save_prediction(pred_mask, output_mask_dir, f'{frame_name}.png')
+            pred_list.append(pred_mask)
+
+            print("processed frame {}, obj_num {}".format(frame_idx, SegTracker.get_obj_num()),end='\r')
+            frame_idx += 1
+        print('\nfinished')
+    
+    ##################
+    # Visualization
+    ##################
+
+    # draw pred mask on frame and save as a video
+    height, width = pred_list[0].shape
+    fourcc =  cv2.VideoWriter_fourcc(*"mp4v")
+
+    out = cv2.VideoWriter(io_args['output_video'], fourcc, fps, (width, height))
+
+    frame_idx = 0
+    for img_path in imgs_path:
+        frame_name = os.path.basename(img_path).split('.')[0]
+        frame = cv2.imread(img_path)
+        frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
+
+        pred_mask = pred_list[frame_idx]
+        masked_frame = draw_mask(frame, pred_mask)
+        masked_pred_list.append(masked_frame)
+        cv2.imwrite(f"{io_args['output_masked_frame_dir']}/{frame_name}.png", masked_frame[:, :, ::-1])
+
+        masked_frame = cv2.cvtColor(masked_frame,cv2.COLOR_RGB2BGR)
+        out.write(masked_frame)
+        print('frame {} writed'.format(frame_name),end='\r')
+        frame_idx += 1
+    out.release()
+    print("\n{} saved".format(io_args['output_video']))
+    print('\nfinished')
+
+    # save colorized masks as a gif
+    imageio.mimsave(io_args['output_gif'], masked_pred_list, fps=fps)
+    print("{} saved".format(io_args['output_gif']))
+
+    # zip predicted mask
+    os.system(f"zip -r {io_args['tracking_result_dir']}/{video_name}_pred_mask.zip {io_args['output_mask_dir']}")
+
+    # manually release memory (after cuda out of memory)
+    del SegTracker
+    torch.cuda.empty_cache()
+    gc.collect()
+
+
+    return io_args['output_video'], f"{io_args['tracking_result_dir']}/{video_name}_pred_mask.zip"