David Shapiro
Patreon: https://www.patreon.com/daveshap (Discord via Patreon)
Substack: https://daveshap.substack.com/ (Free Mailing List)
LinkedIn: https://www.linkedin.com/in/dave-shap-automator/
GitHub: https://github.com/daveshap
Similar Posts
20 thoughts on “Recursively summarize text of any length with GPT-3”
Comments are closed.
thank you so much
thank you 🙂
I'm super not corporate, and/but this seems essential, chunks depend on surrounding chunks for further summarisation 🙂 clearly yes thank you.
this was really cool but doesnt feel economically viable with the token usage no?
Just found your Channel and your Plan with this series. This is super impressive. Thanks for doing it. 💯
I don't know if it was the intention of the video but I wound up reading the whole summary by using . and , to frame by frame though the summary. I guess I know what the book version of Alice in Wonderland is now.
hello, I am afraid we can lose some key concepts maybe while connecting the chunks, can you please share what you think about this worry ?
Have you ever tried improving the summarization by adding the context before and after a chunk? I’ll probably try this myself, but I wonder if the summarization could be improved if you feed the model some “key points” from the text along with every chunk. You can imagine that if you want a high-quality summarization, it would be good for the model to be able to point out things like “what happened to Alice here was foreshadowed when she was doing x.” Or something like that. For more academic texts, it might be something like relating concepts that were introduced in the text at different places in the text. Instead of just giving the definition of concept A, the model includes some additional info about how it relates to concept B even though it was not included in the chunk.
My guess is that you’d have to run a few different prompts across the text (though you could use a language model from huggingface to save some cash for some of the tasks) and then use the outputs as input to the general summarization. Like, use a model to extract the most important concepts from a paper (maybe make use of metadata and such), then store all the concepts in some way so that you can use them during summarization.
Any thoughts on this?
super useful to say the least, thank you so much mate
Why not just paste whatever scientific article in openai playground and summarize? I am just a beginner. Thank you
One of my dreams was to be able to click on parts of a summary and then it would expand it into bigger summaries. (the opposite direction). You would have to know which part in the summary corresponds to the original chunk, but that would be a supercool idea!
When you're breaking into chunks in this fashion, don't you risk cutting in the middle of a word/sentence which can impact summarization of that chunk?
45 iterations of api use? total cost ? just curious 👀 as if you proof read the summarized content. rewrite manually. check and then do the final work. check again and again.
to make it perfect and to stnadard
Works very well! Thank you – excellent tutorial
When Dave brought up that calculator to make a division query my head was brought back down to earth. "I'm really okay," I said to myself.
thanks you
I AM A STUDENT. I WANT TO GET THE PDF BOOKS I HAVE IN HAND IN SUMMARY, CAN YOU PLEASE HELP ME? I DON'T KNOW HOW TO INSTALL THIS SYSTEM, WHAT ARE THE PROCESSING STEPS. I NEED HELP FROM YOU
Thank you for this, I no zero python but this makes me want to know it! I look forward to trying this out on my academic books. Question, have you made an updated video on this process?
I have a case usage of needing 2 million words distilled down to 10k, what / where can I find mass word input for such summerization? Or what is a workaround for me to arrive at this context output desire? I was trying to upload either a PDF and or have a link that I could just copy paste over into chat GPT, but I have the free version and it tells me it can't read links, what can I do? I'm obviously not trying to have the entire 2 million at once, but that is the end game result over time I desire.
is there a quick way of getting reports summarised, like huge pdfs with ton load of data in1 page?
Hi @davidshapiroautomator , thanks for the video. Wondering what is including the the openaiapikey file – is it just the key?