GPT-1 Paper Explained – The Engineering of Conscious Experience

Aladdin Persson

#PaperReview #PaperExplained

Blog post link: https://openai.com/blog/language-unsupervised/
Paper link: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

<iframe></p> <p><a href="https://www.youtube.com/watch?v=m9UCVqd5lGY">Source</a></p> <div class="be1e40beae42d993bafb8643f4ddde8b" data-index="3" style="float: none; margin:10px 0 10px 0; text-align:center;"> <script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script> <ins class="adsbygoogle" style="display:block; text-align:center;" data-ad-layout="in-article" data-ad-format="fluid" data-ad-client="ca-pub-9244112244416304" data-ad-slot="4549240677"></ins> <script> (adsbygoogle = window.adsbygoogle || []).push({}); </script> </div> <div style="font-size: 0px; height: 0px; line-height: 0px; margin: 0; padding: 0; clear: both;"></div> </div> </article> <div class="clearfix"></div> <ul class="default-theme-post-navigation"> <li class="theme-nav-previous"><a href="https://theengineeringofconsciousexperience.com/%d0%be%d0%b1%d0%bd%d0%be%d0%b2%d0%bb%d1%8f%d0%b5%d0%bc-%d0%b3%d0%b0%d1%80%d0%b4%d0%b5%d1%80%d0%be%d0%b1-%d0%ba-%d0%bb%d0%b5%d1%82%d1%83-%d1%81%d1%80%d0%b0%d0%b2%d0%bd%d0%b8%d0%b2%d0%b0%d0%b5%d0%bc/" rel="prev"><span class="meta-nav">←</span> Обновляем гардероб к лету. Сравниваем цены в Киеве и Москве – #25 Украина_НЕ_Россия</a></li> <li class="theme-nav-next"><a href="https://theengineeringofconsciousexperience.com/action-painting-workshop/" rel="next">action painting workshop <span class="meta-nav">→</span></a></li> </ul> <div class="clearfix"></div> <h3 class='comment-reply-title'>Similar Posts</h3> <div class="mb-related-posts mb-simple-featured-posts mb-simple-featured-posts-wrap row"> <article class="mb-featured-article col-md-4 px-lg-3 post"> <a class="post-thumbnail" href="https://theengineeringofconsciousexperience.com/%d0%be%d0%b1%d0%bd%d0%be%d0%b2%d0%bb%d1%8f%d0%b5%d0%bc-%d0%b3%d0%b0%d1%80%d0%b4%d0%b5%d1%80%d0%be%d0%b1-%d0%ba-%d0%bb%d0%b5%d1%82%d1%83-%d1%81%d1%80%d0%b0%d0%b2%d0%bd%d0%b8%d0%b2%d0%b0%d0%b5%d0%bc/" aria-hidden="true" tabindex="-1"> <img width="501" height="282" src="https://theengineeringofconsciousexperience.com/wp-content/uploads/2021/06/1622582712_maxresdefault.jpg" class="attachment-magazinebook-featured-image-medium size-magazinebook-featured-image-medium wp-post-image" alt="" decoding="async" loading="lazy" srcset="https://theengineeringofconsciousexperience.com/wp-content/uploads/2021/06/1622582712_maxresdefault.jpg 1280w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2021/06/1622582712_maxresdefault-300x169.jpg 300w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2021/06/1622582712_maxresdefault-1024x576.jpg 1024w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2021/06/1622582712_maxresdefault-768x432.jpg 768w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2021/06/1622582712_maxresdefault-520x293.jpg 520w" sizes="(max-width: 501px) 100vw, 501px" /> </a> <span class="cat-links"><a href="https://theengineeringofconsciousexperience.com/category/gpt-3/" rel="category tag">GPT 3</a></span> <header class="entry-header"> <h3 class="entry-title"><a href="https://theengineeringofconsciousexperience.com/%d0%be%d0%b1%d0%bd%d0%be%d0%b2%d0%bb%d1%8f%d0%b5%d0%bc-%d0%b3%d0%b0%d1%80%d0%b4%d0%b5%d1%80%d0%be%d0%b1-%d0%ba-%d0%bb%d0%b5%d1%82%d1%83-%d1%81%d1%80%d0%b0%d0%b2%d0%bd%d0%b8%d0%b2%d0%b0%d0%b5%d0%bc/" rel="bookmark">Обновляем гардероб к лету. Сравниваем цены в Киеве и Москве – #25 Украина_НЕ_Россия</a></h3> <div class="entry-meta"> <span class="posted-on"><i class="far fa-calendar-alt"></i><a href="https://theengineeringofconsciousexperience.com/%d0%be%d0%b1%d0%bd%d0%be%d0%b2%d0%bb%d1%8f%d0%b5%d0%bc-%d0%b3%d0%b0%d1%80%d0%b4%d0%b5%d1%80%d0%be%d0%b1-%d0%ba-%d0%bb%d0%b5%d1%82%d1%83-%d1%81%d1%80%d0%b0%d0%b2%d0%bd%d0%b8%d0%b2%d0%b0%d0%b5%d0%bc/" rel="bookmark"><time class="entry-date published updated" datetime="2021-05-23T09:27:28-07:00">May 23, 2021</time></a></span><span class="byline"><i class="far fa-user-circle"></i><span class="author vcard"><a class="url fn n" href="https://theengineeringofconsciousexperience.com/author/klymenko_time/">Klymenko Time</a></span></span> </div> </header> </article> <article class="mb-featured-article col-md-4 px-lg-3 post"> <a class="post-thumbnail" href="https://theengineeringofconsciousexperience.com/how-to-identify-the-partition-style-gpt-or-mbr/" aria-hidden="true" tabindex="-1"> <img width="501" height="282" src="https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/11/1606249675_maxresdefault.jpg" class="attachment-magazinebook-featured-image-medium size-magazinebook-featured-image-medium wp-post-image" alt="" decoding="async" loading="lazy" srcset="https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/11/1606249675_maxresdefault.jpg 1280w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/11/1606249675_maxresdefault-300x169.jpg 300w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/11/1606249675_maxresdefault-1024x576.jpg 1024w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/11/1606249675_maxresdefault-768x432.jpg 768w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/11/1606249675_maxresdefault-520x293.jpg 520w" sizes="(max-width: 501px) 100vw, 501px" /> </a> <span class="cat-links"><a href="https://theengineeringofconsciousexperience.com/category/gpt-3/" rel="category tag">GPT 3</a></span> <header class="entry-header"> <h3 class="entry-title"><a href="https://theengineeringofconsciousexperience.com/how-to-identify-the-partition-style-gpt-or-mbr/" rel="bookmark">How to Identify the Partition style GPT or MBR</a></h3> <div class="entry-meta"> <span class="posted-on"><i class="far fa-calendar-alt"></i><a href="https://theengineeringofconsciousexperience.com/how-to-identify-the-partition-style-gpt-or-mbr/" rel="bookmark"><time class="entry-date published updated" datetime="2020-11-10T07:52:17-07:00">November 10, 2020</time></a></span><span class="byline"><i class="far fa-user-circle"></i><span class="author vcard"><a class="url fn n" href="https://theengineeringofconsciousexperience.com/author/84049d21a1643587fa736cb96f9468e6/">Innovative Computer Solutions</a></span></span> </div> </header> </article> <article class="mb-featured-article col-md-4 px-lg-3 post"> <a class="post-thumbnail" href="https://theengineeringofconsciousexperience.com/ibadah-online-gpt-bukit-horeb-ibadah-doa-semalam-suntuk-3-kamis-9-12-2020-pkl-02-30-wib/" aria-hidden="true" tabindex="-1"> <img width="501" height="282" src="https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/12/1607635442_maxresdefault.jpg" class="attachment-magazinebook-featured-image-medium size-magazinebook-featured-image-medium wp-post-image" alt="" decoding="async" loading="lazy" srcset="https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/12/1607635442_maxresdefault.jpg 1280w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/12/1607635442_maxresdefault-300x169.jpg 300w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/12/1607635442_maxresdefault-1024x576.jpg 1024w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/12/1607635442_maxresdefault-768x432.jpg 768w, https://theengineeringofconsciousexperience.com/wp-content/uploads/2020/12/1607635442_maxresdefault-520x293.jpg 520w" sizes="(max-width: 501px) 100vw, 501px" /> </a> <span class="cat-links"><a href="https://theengineeringofconsciousexperience.com/category/gpt-3/" rel="category tag">GPT 3</a></span> <header class="entry-header"> <h3 class="entry-title"><a href="https://theengineeringofconsciousexperience.com/ibadah-online-gpt-bukit-horeb-ibadah-doa-semalam-suntuk-3-kamis-9-12-2020-pkl-02-30-wib/" rel="bookmark">IBADAH ONLINE GPT Bukit Horeb – Ibadah DOA SEMALAM SUNTUK 3| KAMIS 9.12.2020 | PKL. 02.30 WIB.</a></h3> <div class="entry-meta"> <span class="posted-on"><i class="far fa-calendar-alt"></i><a href="https://theengineeringofconsciousexperience.com/ibadah-online-gpt-bukit-horeb-ibadah-doa-semalam-suntuk-3-kamis-9-12-2020-pkl-02-30-wib/" rel="bookmark"><time class="entry-date published updated" datetime="2020-12-10T02:04:51-07:00">December 10, 2020</time></a></span><span class="byline"><i class="far fa-user-circle"></i><span class="author vcard"><a class="url fn n" href="https://theengineeringofconsciousexperience.com/author/142493dcab790525d3beb6b420f9dfc3/">GPT Bukit Horeb Palangka Raya</a></span></span> </div> </header> </article> </div> <div id="comments" class="comments-area"> <h5 class="comments-title"> 7 thoughts on “<span>GPT-1 Paper Explained</span>” </h5> <ol class="comment-list"> <li id="comment-235217" class="comment even thread-even depth-1"> <article id="div-comment-235217" class="comment-body"> <footer class="comment-meta"> <div class="comment-author vcard"> <b class="fn"><a href="https://www.youtube.com/channel/UCkzW5JSFwvKRjXABI-UTAkQ" class="url" rel="ugc external nofollow">Aladdin Persson</a></b> <span class="says">says:</span> </div> <div class="comment-metadata"> <a href="https://theengineeringofconsciousexperience.com/gpt-1-paper-explained/#comment-235217"><time datetime="2021-05-23T09:42:56-07:00">May 23, 2021 at 9:42 am</time></a> </div> </footer> <div class="comment-content"> <p>I saw some people wanted from scratch implementations & particularly on previous BERT video. Implementing these (correctly) takes a lot of time so it's not always worth the time investment so we'll see if I'll do it.. Right now I'm focusing more on Kaggle and will have more videos on that soon 🙂</p> </div> </article> </li> <li id="comment-235223" class="comment odd alt thread-odd thread-alt depth-1"> <article id="div-comment-235223" class="comment-body"> <footer class="comment-meta"> <div class="comment-author vcard"> <b class="fn"><a href="https://www.youtube.com/channel/UCCA9ydw72lJosY2aioE2c_A" class="url" rel="ugc external nofollow">Abhishek Maity</a></b> <span class="says">says:</span> </div> <div class="comment-metadata"> <a href="https://theengineeringofconsciousexperience.com/gpt-1-paper-explained/#comment-235223"><time datetime="2021-05-23T09:46:36-07:00">May 23, 2021 at 9:46 am</time></a> </div> </footer> <div class="comment-content"> <p>So are you going to put more video on solving the Kaggle problem Use cases??(ML or NLP using DL) ?</p> </div> </article> </li> <li id="comment-235222" class="comment even thread-even depth-1"> <article id="div-comment-235222" class="comment-body"> <footer class="comment-meta"> <div class="comment-author vcard"> <b class="fn"><a href="https://www.youtube.com/channel/UCFCMgGCNJ0bWhYMlvJp2RYQ" class="url" rel="ugc external nofollow">Tejasvi</a></b> <span class="says">says:</span> </div> <div class="comment-metadata"> <a href="https://theengineeringofconsciousexperience.com/gpt-1-paper-explained/#comment-235222"><time datetime="2021-05-23T10:39:57-07:00">May 23, 2021 at 10:39 am</time></a> </div> </footer> <div class="comment-content"> <p>I like your paper explanation videos very much, they helped me a lot in reading paper as i am a newbie , Would love to see more videos , Thanx a lot</p> </div> </article> </li> <li id="comment-235221" class="comment odd alt thread-odd thread-alt depth-1"> <article id="div-comment-235221" class="comment-body"> <footer class="comment-meta"> <div class="comment-author vcard"> <b class="fn"><a href="https://www.youtube.com/channel/UCse-i3PNddq1vu0xswnjrCw" class="url" rel="ugc external nofollow">Garik Hakobyan</a></b> <span class="says">says:</span> </div> <div class="comment-metadata"> <a href="https://theengineeringofconsciousexperience.com/gpt-1-paper-explained/#comment-235221"><time datetime="2021-05-23T10:42:44-07:00">May 23, 2021 at 10:42 am</time></a> </div> </footer> <div class="comment-content"> <p>Good work. Waiting for new videos</p> </div> </article> </li> <li id="comment-235220" class="comment even thread-even depth-1"> <article id="div-comment-235220" class="comment-body"> <footer class="comment-meta"> <div class="comment-author vcard"> <b class="fn"><a href="https://www.youtube.com/channel/UCsxz5maSJlG7KvluKvTUVJQ" class="url" rel="ugc external nofollow">Stephen Fernandes</a></b> <span class="says">says:</span> </div> <div class="comment-metadata"> <a href="https://theengineeringofconsciousexperience.com/gpt-1-paper-explained/#comment-235220"><time datetime="2021-05-24T09:32:55-07:00">May 24, 2021 at 9:32 am</time></a> </div> </footer> <div class="comment-content"> <p>Hey BERT and GPT1 use sentencepice representation of sentences while training , but how do they fine tune and work with POS tagging task as they require to classify a word based on the enter word per POS token ?? any idea how its done ???</p> </div> </article> </li> <li id="comment-235219" class="comment odd alt thread-odd thread-alt depth-1"> <article id="div-comment-235219" class="comment-body"> <footer class="comment-meta"> <div class="comment-author vcard"> <b class="fn"><a href="https://www.youtube.com/channel/UCA6chhF0KcnTYaWHSuvI7rg" class="url" rel="ugc external nofollow">RoG 007</a></b> <span class="says">says:</span> </div> <div class="comment-metadata"> <a href="https://theengineeringofconsciousexperience.com/gpt-1-paper-explained/#comment-235219"><time datetime="2021-05-24T12:51:46-07:00">May 24, 2021 at 12:51 pm</time></a> </div> </footer> <div class="comment-content"> <p>So are you planning to code GPT from scratch ?</p> </div> </article> </li> <li id="comment-235218" class="comment even thread-even depth-1"> <article id="div-comment-235218" class="comment-body"> <footer class="comment-meta"> <div class="comment-author vcard"> <b class="fn"><a href="https://www.youtube.com/channel/UCdpy1v7Br6n0RTQutR3cW7Q" class="url" rel="ugc external nofollow">Jeff W</a></b> <span class="says">says:</span> </div> <div class="comment-metadata"> <a href="https://theengineeringofconsciousexperience.com/gpt-1-paper-explained/#comment-235218"><time datetime="2021-05-24T17:14:17-07:00">May 24, 2021 at 5:14 pm</time></a> </div> </footer> <div class="comment-content"> <p>Great video! If you're able to do the GPT-2 and GPT-3 papers as well that would be great, but this is a solid start to understanding how GPT differs from transformers.</p> <p>I tend to bounce off of the dry academic papers; sometimes it feels like some of them don't <b>want</b> to be understood. But your explainer videos have really helped me get a grip on the concepts so I can go back to the paper and get more out of the details.</p> </div> </article> </li> </ol> <p class="no-comments">Comments are closed.</p> </div> </main> </div> <div class="col-md-3 px-lg-3 "> </div> </div> </div> </div> <footer id="colophon" class="site-footer"> <div class="container"> <div class="row"> <div class="col-md-12 text-center"> <div class="site-info"> <span> Powered By: <a href="https://wordpress.org/" target="_blank">WordPress</a> </span> <span class="sep"> | </span> <span> Theme: <a href="https://odiethemes.com/themes/magazinebook/" target="_blank">MagazineBook</a> By OdieThemes </span> </div> </div> </div> </div> </footer> </div> <script>(function(){var advanced_ads_ga_UID="UA-88163215-1",advanced_ads_ga_anonymIP=!!1;window.advanced_ads_check_adblocker=function(t){var n=[],e=null;function a(t){var n=window.requestAnimationFrame||window.mozRequestAnimationFrame||window.webkitRequestAnimationFrame||function(t){return setTimeout(t,16)};n.call(window,t)}return a((function(){var t=document.createElement("div");t.innerHTML=" ",t.setAttribute("class","ad_unit ad-unit text-ad text_ad pub_300x250"),t.setAttribute("style","width: 1px !important; height: 1px !important; position: absolute !important; left: 0px !important; top: 0px !important; overflow: hidden !important;"),document.body.appendChild(t),a((function(){var a,o,i=null===(a=(o=window).getComputedStyle)||void 0===a?void 0:a.call(o,t),d=null==i?void 0:i.getPropertyValue("-moz-binding");e=i&&"none"===i.getPropertyValue("display")||"string"==typeof d&&-1!==d.indexOf("about:");for(var c=0,r=n.length;c<r;c++)n[c](e);n=[]}))})),function(t){"undefined"==typeof advanced_ads_adblocker_test&&(e=!0),null!==e?t(e):n.push(t)}}(),(()=>{function t(t){this.UID=t,this.analyticsObject="function"==typeof gtag;var n=this;return this.count=function(){gtag("event","AdBlock",{event_category:"Advanced Ads",event_label:"Yes",non_interaction:!0,send_to:n.UID})},function(){if(!n.analyticsObject){var e=document.createElement("script");e.src="https://www.googletagmanager.com/gtag/js?id="+t,e.async=!0,document.body.appendChild(e),window.dataLayer=window.dataLayer||[],window.gtag=function(){dataLayer.push(arguments)},n.analyticsObject=!0,gtag("js",new Date)}var a={send_page_view:!1,transport_type:"beacon"};window.advanced_ads_ga_anonymIP&&(a.anonymize_ip=!0),gtag("config",t,a)}(),this}advanced_ads_check_adblocker((function(n){n&&new t(advanced_ads_ga_UID).count()}))})();})();</script><div style="clear:both;width:100%;text-align:center; font-size:11px; "><a target="_blank" title="WP2Social Auto Publish" href="https://xyzscripts.com/wordpress-plugins/facebook-auto-publish/compare" >WP2Social Auto Publish</a> Powered By : <a target="_blank" title="PHP Scripts & Programs" href="http://www.xyzscripts.com" >XYZScripts.com</a></div><script type="text/javascript" src="https://theengineeringofconsciousexperience.com/wp-content/themes/magazinebook/js/navigation.js?ver=1.0.9" id="magazinebook-navigation-js"></script> <script type="text/javascript" src="https://theengineeringofconsciousexperience.com/wp-content/themes/magazinebook/js/skip-link-focus-fix.js?ver=1.0.9" id="magazinebook-skip-link-focus-fix-js"></script> <script type="text/javascript" src="https://theengineeringofconsciousexperience.com/wp-content/themes/magazinebook/js/jquery.easy-ticker.js?ver=3.1.0" id="magazinebook-news-ticker-js"></script> <script type="text/javascript" src="https://theengineeringofconsciousexperience.com/wp-content/themes/magazinebook/js/splide.min.js?ver=2.3.1" id="splide-js-js"></script> <script type="text/javascript" src="https://theengineeringofconsciousexperience.com/wp-content/themes/magazinebook/js/theme.js?ver=1.0.9" id="magazinebook-theme-js-js"></script> <script>!function(){window.advanced_ads_ready_queue=window.advanced_ads_ready_queue||[],advanced_ads_ready_queue.push=window.advanced_ads_ready;for(var d=0,a=advanced_ads_ready_queue.length;d<a;d++)advanced_ads_ready(advanced_ads_ready_queue[d])}();</script> </body> </html>