Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
sbszllr
on July 24, 2023
|
parent
|
context
|
favorite
| on:
Attention Is Off By One
The family of sigmoid functions has nice gradient properties with theoretical backing. Good starting read:
https://stats.stackexchange.com/questions/162988/why-sigmoid...
lostmsu
on July 25, 2023
[–]
Doesn't really explain why not more typical ReLU + layer norm or some alternative.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
