In recent years there has been an explosion of methods based on self-attention and in particular Transformers, first in the field of Natural Language Processing and recently also in the field of Computer Vision.
If you don’t know what Transformers are, or if you want to know more about the mechanism of self-attention, I suggest you have a look at my first article on this topic.
The success of the transformers is related to their extreme effectiveness and ability to solve non-trivial problems in a superior way compared to previous architectures such as RNNs in natural language processing or convolutional networks in computer vision. At the basis of Transformers there is and always has been the mechanism of attention, considered “all you need”, indispensable and the true beating heart of this architecture. But not all that glitters is gold, in fact, the calculation of self-attention brings with it huge computational and memory costs that, such as require very large amounts of video memory and cause high training times.
This has not gone unnoticed by big companies like Apple and Google, who have been working hard to make Transformers that are not only able to achieve state-of-the-art results but also do so efficiently.
If this in-depth educational content is useful for you, subscribe to our AI research mailing list to be alerted when we release new material.
Efficient Transformers
Recently, Lukasz Kaiser, one of the co-creators of Transformers and Google’s researcher, presented a series of improvements to make Transformers more efficient even maintaining the self-attention mechanism, and the first and probably one of the most important aspect he focused on was memory efficiency.
Transformers use a large amount of memory, as multiple intermediate tensors are created and maintained during their execution, and as they accumulate they quickly saturate the video memory in the absence of large resources.
The method proposed by Google Brain to get around this problem is to avoid keeping all the tensors in memory, but rather to make each step of the process reversible.
To do this, two tensors are maintained at each step, the one resulting from the application of the layer and a copy of the previous tensor. This allows the process to continue without having to maintain the entire chain of intermediate tensors, but only those of the last step.
With this stratagem, one is therefore able to significantly reduce memory costs while still achieving the same results as a normal transformer. This could be one of the smartest methods currently known to keep the transformer architecture completely based on self-attention as it has traditionally been, but at a lower cost.
AFT: Attention Free Transformers
But at what point does the calculation of self-attention become so complex to manage? Is there a way to eliminate quadratic complexity? Do we really need attention calculation as we are currently using it? These are the questions that Apple’s researchers have asked themselves, and which form the basis of the Attention Free Transformers.
The problem lies in the dot product, which is used to combine queries, keys and values and is done by considering every single input vector as a query. Aware of this, the Attention Free Transformer is designed to never dot product while retaining the benefits.
As in the original transformer, AFT initially creates Q, K and V as a result of the linear transformation of input with the matrices of queries, keys and values.
The peculiarity is that at this point, instead of carrying out the dot product for the creation of the attention matrix, a weighted average of the values is carried out for each target position. The result of this is combined with the query by means of element-wise multiplication.
Through this mechanism, it is possible to obtain a linear computational and space complexity dependent on the number of output features and the length of the sequence considered. Conceptually, it is simply a different way of making information flow within the sequence but in a much less expensive way.
By testing the Attention Free Transformer on many tasks previously tested in the literature with the original Transformer, it was possible to see how, for example in the case of the Vision Transformer shown in the figure (left), the features obtained from the AFT (right), in this case, the AFT-Conv version, seems to be still meaningful even if approximated.
With this mechanism, it was not only possible to cut the cost of calculating attention but excellent results were obtained in all the tasks considered, an indication that this solution is able to maintain all the advantages of the dot product without the costs it requires to be calculated.
FNet: Fourier Networks
But there are also those who have considered abandoning the calculation of attention altogether and have gone in search of a mechanism that can be as effective as attention but not as costly to calculate.
It would seem that a good candidate for this task would be the Fourier transform. The Fourier transform does nothing more than taking a function in one domain, e.g. time, and take it into another domain, e.g. frequency.
The Fourier Network proposed by Google and based on that mechanism is exactly the same as a normal Transformers but with the attention calculation block replaced by a layer that takes care of the Fourier transform.
Having N input vectors, composed of T tokens, the Fourier transform is first applied to what is called the “hidden domain” and then to the “sequence domain”. All this is done without any type of parameter, bringing with it enormous advantages, since the parameters present in the other layers are the only ones that can be trained, thus reducing the number of model parameters.
So the transformation is linear and is applied to the input first column-wise and then row-wise. In addition to the simplicity obtained by applying the Fourier transform, there is also the advantage that it is reversible.
Exactly as in the case of AFT, this series of transformations makes it possible for the various parts of the sequence to influence each other and the result is a transformed representation that contains information derived from the various parts of the input sequence.
Apparently, this seems to be a very interesting method that is able to significantly reduce the costs of attention calculation and achieve discrete results. In fact, the FNet does not seem to be better than the classical Transformers and probably there are other better methods to obtain comparable results at lower costs but in the absence of large computational resources, the FNet may be a really valid choice.
What awaits us in the future?
It is now clear that the Transformer is an enormously powerful architecture capable of solving the most diverse problems with results we have never seen before, from translation to segmentation to classification. However, these have for too long been linked to their exaggerated consumption of resources, and their arrival in the world of computer vision has highlighted this problem even more and prompted many researchers to seek solutions.
In the future, we may see transformers based on attention but optimized to be lighter, or transformers deprived of their attention mechanism to make room for more approximate techniques, or totally new networks, similar to transformers, but with different input transformations strategies such as the Fourier transform. And if you want to be ready and know more about them I suggest you read my articles on Transformers and on DINO.
One thing is certain, if for the moment it has been impossible for most people to make full use of this architecture, it will soon be available to everyone, and its great potential combined with accessibility will make transformers even more pervasive and central than they already are.
References and Insights
[1] “Lukasz Kaiser”. “Efficient Transformers”
[2] “Shuangfei Zhai et al”. “An Attention Free Transformer”
[3] “James Lee-Thorp et al”. “FNet: Mixing Tokens with Fourier Transforms”
[4] “Yannic Kilcher”. “FNet: Mixing Tokens with Fourier Transforms (Machine Learning Research Paper Explained)”
[5] “Davide Coccomini”. “On Transformers, Timesformers and Attention”
[6] “Davide Coccomini”. “On DINO, Self-Distillation with no labels”
This article was originally published on Towards Data Science and re-published to TOPBOTS with permission from the author.
Enjoy this article? Sign up for more AI updates.
We’ll let you know when we release more technical education.
wqszpwdrl says
Is Attention What You Really Need In Transformers?
awqszpwdrl
[url=http://www.gxj9ur8p5788771939d6nxbcw9i1a9sbs.org/]uwqszpwdrl[/url]
wqszpwdrl http://www.gxj9ur8p5788771939d6nxbcw9i1a9sbs.org/
Custom Design Plain Custom Color Polyester Fit Blank Men Golf Polo T Shirts Customizable For Corporate Events says
Pure Golf Polo Tshirts For Men High-quality Men's Polo T-shirt Embroidered Logo Knit Fabric Ideal For Golf Wear
Pneumatic Punching Machine
Automatic Punching Machine
Women's White Hoodie And Jogger Set Casual Oversized Sweatshirt Comfortable Cotton Loungewear High-quality Winter Clothes
Motorcoach
Sleeper coach
Durable And Comfortable Knit Custom T Shirts For Men Cotton Polo Shirts Custom Logo Cotton Mens Polo
Automatic Punching Machine
Plastic Window Machine
Women's Light Blue Polo Shirt Custom Logo Cotton Comfortable Fitted Shirt Fashion Trim Sports Or Casual Wear.
http://www.tbgfrisbee.no
UL1007 PVC Wire
Tour Coach Auto
Pvc Door Manufacturing Machine
Ul1007 Pvc Electronic Wire
Custom Design Plain Custom Color Polyester Fit Blank Men Golf Polo T Shirts Customizable For Corporate Events
Electric Car Vehicle says
Men's Basic Heavyweight Hoodie
Mini Electric Cars For Adult
Rotary Switch
Oversized Drop Shoulder Men's Hoodie
http://www.setsatian.ac.th
Speed Shift Cam Operated Limit Switch
Electric Convertible Car
Men's Zip Up Hoodie With Tribal Sleeve Patterns Streetwear Cotton Sweatshirt
Tesla Model 3 Electric Car
Custom Plus Size Hoodie For Men
Electric Car Batteries
Cam Switch
Limit Switch
Drop Shoulder Men's Hoodie
Changeover Switch
Electric Car Vehicle
Transport Forwarder says
http://www.pawilony.biz.pl
High Quality 3d Puff Print Pullover Ribbed Cuffs And Hem Side Pockets For Convenience Adjustable Hood
1200Y Rotary Wood CNC
Adjustable Hood With Drawstrings Perfect For Casual Outfits And Sports Available In Plus Sizes Hoodie Tops
Woodworking panel saw
Agent Shipping China
Agent Shipping China
Women's Fleece Hoodie With Slogan Print – Ultra-soft And Warm Comfortable Oversized Fit Adjustable Drawstring Hood
Chic Women's Cropped Sweatshirt Set – Comfy Loungewear Trendy Crop Top Design Soft Cotton Material
Consigning Of Freight
Agent Cargo
CO2 laser machine
China Factory New Digital Printing Pattern Round Neck Casual Over Size Casual Fashion T Shirt For Women
Aluminum strip sawing
4 axis wood cnc router
Transport Forwarder
www.rwopr.pl says
Eco-friendly Women's Hoodie With Adorable Cat Illustration Soft Fabric For Lounge Or Streetwear
Ovation DCS
Women's Hooded Sweatshirts In Assorted Colors Soft Fleece Pullover Vintage Chicago Print Athletic Casual Winter Clothes
GE FANUC PLC
Casual Athletic Zip-up Hoodie Alls Play Surf Motif For Women Durable Cotton Blend Ideal For Sports And Everyday Fashion
Lighted Heliport
ICS Triplex PLC
Obstruction Lamp
Two-piece Set Green Cropped Hoodie With Matching Shorts Women's Sportswear Cotton French Terry Loungewear Summer Outfit
Helipad Perimeter Lighting
Black Custom Multi-logo Printed Hoodie Women's Fashion Pullover High-quality Cotton Streetwear Sweatshirt With Cropped Fit
HIMA PLC Modules
Aircraft Obstruction Lights
HONEYWELL DCS
Low Intensity Aircraft Warning Light
http://www.rwopr.pl
Geely says
Trendy Green Men's Hoodie With Contrast Logo Soft Cotton Blend Hooded Top For Casual Or Athleisure Wear With Snug Ribbed Hem
Pvc Tarpaulin
Snow Tarp
Volkswagen
Used Car Exports
GAC Trumpchi
Men's Purple Full Zip Hoodie With Green Accent Graphics Comfortable Cotton Blend Streetwear Hooded Sweatshirt
Classic Green Fleece Hoodie For Men With Zippered Pockets Cozy Cotton Blend For Running
Clear Tarpaulin
Urban Techwear Men's Olive Green Hoodie With Multiple Pockets Comfortable Heavyweight Cotton Perfect For Casual Streetwear
Truck Tarpaulin Cover
plot28.com
Fang Chen Bao
Clear Tarpaulin
Men's White Lace-up Hoodie 2024 Trendsetting Sweatshirt With Custom Logo Relaxed Fit Cozy Pullover Streetwear Top
Geely
auto european type micro relay says
Metal Wire Mesh Screen
sketch book spiral book
Crimped Wire Mesh
renobeya.com
Metal Wire Mesh Sheets
8pcs washable crayon kids baby bath time paints drawing pens
What are the advantages of solar LED lawn lights
Trendy Women's Winter Hoodie With Bold Graphic Print Comfortable Pullover For Fashion Enthusiasts
Stylish Women's Black And White T-shirts High-quality Cotton Comfort Fit Essential Casual Streetwear Tops Soft
Hydraulic Filter Element By Size
Eco-friendly Bamboo Tshirt Women Soft Breathable Fabric Women's Sustainable Fashion Tee V-neck Design Wholesale Variety
Copper Wire Mesh Factory
Women Croptop Tshirt Cotton Short Sleeve Casual Ladies Fashion Streetwear Oversize Tshirt Woman Customizable For Wholesale
Wholesale Women's Blank Tshirt Custom Logo Printing Available High-quality Fabric Assorted Color Choices Bulk Order For Branding
double head vertical sawmill
auto european type micro relay
Yoga Bra says
Scarf
Toned T-shirts For Women Customizable Logo High-quality Cotton Streetwear Casual Fit Soft Fabric Versatile Fashion
Brass Screw Shank Ro Water Faucet 1 2
What are the principles of gift box packaging
Colorful Hoodie With Brightly Colored Logo On Top For Urban Fashion Lovers A Comfortable Fitted Street Casual Top
Wholesale Custom Logo Women's T-shirt Manufactured Soft Cotton Fabric Perfect For Business Merchandise
Custom printed Transparent Bobo Balloons
http://www.winsta.jp
How to place ceramic vases in the living room
Chic Asymmetrical Crop Top In Cotton Long Sleeve Comfort Fit One-shoulder Design Women's Fashionable Streetwear
Running Shirts For Women Dry Fit
LC1 D115 3P 220V ac contactor
Scarf
One Piece Swimsuit
Solid Color Hoodies For Women Casual Pullover Fleece Sweatshirt Red Yellow Cyan Blue Ivory Black Pink
Yoga Bra
Street Style Heavyweight Hoodie says
Excavator Arm For Construction
UF Water Filter
Excavator Diamond-Tipped Arm
Commercial Water Filter
accentdladzieci.pl
RO Water Filter
Men's Hoodie With Bold Custom Logo Text Front Statement Piece
Water Dispenser
Custom Logo Black Hoodie Men's Oversized Cotton Streetwear
Ripper Hook For Excavator
Excavator Tunnel Arm
Pre Water Filter
Customizable Solid Color Hoodie For Men
Excavator Arm For Quarrying
Thick Cotton Streetwear Essential
Unisex Fit
Pure electric minivans says
tukurou.club
Minavan
Customized Butterfly Print Hoodies Women Soft Warm Loose Casual Sportswear Female Sweatshirt Fleece Ladies Clothes
Snowy Pine Trees
Snowy Pine Trees
Pure electric minivan
New energy minivan
Fashion Autumn And Winter Women Personalized Long Sleeve Hoodie Casual Letter Embroidery Loose Overcoat
Customized Logo Women Fleece Long Sleeve Loose Hoodies Jacket Fashion Hoody Cropped Woman Hoodie
Fashion Long Sleeve Tops Ladies Streetwear Fleece Oversized Hoodies Around The Neck Oversize Hoodie For Woman
Wholesale Hooded Colorful Fleece Casual Winter Clothing Oversized Women's Hoodies Sweatshirts Hoodie Heavyweight
Bio Ceramic Ring
Pride Christmas Tree
Cotton Ball Garland
New energy minivans
Pure electric minivans
Pu Finger Reading Stick says
Telescopic Handheld Flagpoles
Module Steel Hook
Retractable Shoe Horn
Men's Basic Black Hoodie Durable Soft Cotton Heavyweight Casual Pullover Streetwear Essential Cozy Warm Comfort Fit
http://www.hunin-diary.com
Convenient Chest Pocket Soft Touch Fabric Ideal For Custom Logo Available For Bulk Purchase With Men
Custom Logo For Men Cotton Comfort Subtle Branding All Kinds Of Routine Essential Wardrobe Casual Or Layering
Telescopic Tour Guide Flagpole
Needle For Computerized Flat Knitting Machine And Circular Machine
Telescopic Car Windshield Cleaning Brush
Wholesale Perfect For Casual Or Active Wear Durable Crew Neck Plain Solid Color High Quality Essential Available In Various Size
Women's Casual Cotton Oversized Hoodie Pullover Streetwear Comfort Sweatshirt Available In Chic Colors Custom Logo
For China Quartz Watch Motor
Most Precise Watch Movement
Mini Stamping Part
Pu Finger Reading Stick
Food Grade Non Dairy Creamer says
Custom Logo Tie-dye Hoodies Unisex High Quality Anti-shrink Sweatshirts For Men 2024 Vintage Style Streetwear
Glitter Yarn
Dairy Free Coffee Creamer
Black Cotton Hoodie Men 2024 Custom Logo Floral Sleeve Embellishment High-quality Streetwear Essentials Sweatshirt
sportedzoitanfolyam.hu
Cold Soluble Non Dairy Creamer
Pink Oversized Hoodie For Men 2024 With Unique Custom Logo High-quality Cotton Streetwear Sweatshirt Men's Summer Fashion Top
Women's Cotton Oversized Hoodie Soft Streetwear Warm Y2k Inspired Loungewear Pullover Customizable In Various Chic Colors
Knitting Add Yarn
Sewing Thread
Cold Water Soluble Non Dairy Creamer For Cold Drink
Best Crocheting Yarn
Knitting And Weaving
Non Dairy Coffee Creamer for Cappuccino Foaming Creamer
Manufacturer Wholesale Loose Fit Ladies Shirt Custom Logo Women Oversized T Shirts Soft 100% Cotton Tshirt For Girls
Food Grade Non Dairy Creamer
Wholesale High Quality Cotton High Street Oversized Summer T Shirt For Men Plus Size Mens T Shirts says
Custom Women's T-shirt Long Drop-shoulder Women's T-shirt Wholesale Long-sleeved Casual Short Sleeve
Custom Tote Bags For Business
First Coffee Bags
Custom China High Quality 95% Cotton 5% Spandex Soft Sports Round Neck Curve Bottom Gym Tee Mens T Shirts
http://www.borisevo.myjino.ru
HDPE Hollow Wall Winding Pipe Machine
Fashion Gradient Round Neck European And American Summer Men's Casual Sports T-shirt Short Sleeve Men's Sweatshirt
Rx Cookie Bags
Quenching System Machine Equipment for Aluminum Extrusion Profile
HDPE Hollow Spiral Pipe Machine
Plus Size Organic Plain T Shirts For Men Wholesale High Quality Drop Shoulder Acid Washed Faded Mens T Shirts
Custom Euro Tote Bags
Automatic Handling System with Single-Operator Straightener
Plastic Corrugated Pipe Machine
Pet Plastic Film
Wholesale High Quality Cotton High Street Oversized Summer T Shirt For Men Plus Size Mens T Shirts
Letter Print Sweatshirt Fashion Brand Women's Hoodies New Spring Autumn Casual Hoodies Sweatshirts Top Hoodies says
Cake Box With Handle
Excavator Diesel Fuel Filter 1R-0750
Truck Diesel Engine Fuel Filter for Volvo Truck Heavy Truck
Artridge Oil Fuel Filter for Fuel Pump Dispenser
Custom Printed Womens Drop Shoulder Hoodies High Quality Anti-shrink Sweaters And Hoodies For Women Cotton
Paper Hot Soup Cup Bowl Food Cup
Fuel Filter 31922-2e900 for Korean Car
Holder Paper Coffee Cup
Milk Tea Cup Lid
Truck Spare Engine Parts Fuel Filter Bw5073
http://www.radioklub.blansko.net
Customized Butterfly Print Hoodies Women Soft Warm Loose Casual Sportswear Female Sweatshirt Fleece Ladies Clothes
Ice Cream Cup With Spoon
Fashion Ins Fleece-lined Hoodie Women Loose Spring And Autumn Print Hoodies Oversize Harajuku Thickened Idle Sle Top
Custom Printed Hoodies Oem High Quality Oversized Winter Long Sustainable Polyester Hoodies For Women
Letter Print Sweatshirt Fashion Brand Women's Hoodies New Spring Autumn Casual Hoodies Sweatshirts Top Hoodies