[dtwaling]

P.S.: I am working on a modified finetune.py to see if I can get the advanced convert_archive.py script to work while still using unsloth.  It is a little trickier and more involved than I anticipated, but we shall see.

Right now the main issue I keep running into (which appears to be fairly common) is that I am running into memory resource constraints, and reducing num_stages is not as straight forward as some recommendations suggest.

I have 64GB RAM, but of course the process errors, stating a requirement of ~92GB …a touch more than what I have installed, lol

[dtwaling]

Hi Brian,
I’m giving this post one more try, so hopefully it shows up this time…

Excellent How-to, thank you!
Here are a couple notes –

On Linux it is typical for the “user” instance of Python to be different from the OS’s native instance, which is important to keep separate.  So to use the correct (“user”) instance you need to initialize it in a py virtual env.
– e.g.: if you have Python3 installed at ~/anaconda3, then you first need to run `source /home/dtdubs/anaconda3/bin/activate
conda init`

One correction at the end of Step 3, for the advanced option when using convert_archive.py:

The input parameters should be hyphenated instead of underscores, and the output format param is now “oai” – using “jsonl” produces no output.
Corrected cmd: py convert_archive.py --archive-path . --output-dir . --output-formats oai

And, lastly, the current X archive export uses subdirectories, which can have a very large number of folders.  So, I have made a small alteration to your scripts to specify the data sub-directory – e.g.: in prepare_data.py(line 6) the path param for the open call is now for open('data/tweets.js', ....  This way I can keep my(your) scripts separate from the clutter in the top level archive folder.

That is all for now – so far so good!  🙂

[dtwaling]

Weird, I tried to post a response twice and it is not showing up here …?
I checked my email to see if there might be any verification step I missed, but nothing.
I have a couple notes on the how-to so hopefully if this gets corrected I will definitely share them, but I don’t want to spam this discussion topic.

-D

[dtwaling]

Hi Brian,

Excellent How-to, thank you!
Here are a couple notes –

On Linux it is typical for the “user” instance of Python to be different from the OS’s native instance, which is important to keep separate.  So to use the correct (“user”) instance you need to initialize it in a py virtual env.
– e.g.: if you have Python3 installed at ~/anaconda3, then you first need to run `source /home/dtdubs/anaconda3/bin/activate
conda init`

One correction at the end of Step 3, for the advanced option when using convert_archive.py:

The input parameters should be hyphenated instead of underscores.
Corrected cmd: py convert_archive.py --archive-path . --output-formats jsonl

And, lastly, the current X archive export uses subdirectories, which can have a very large number of folders.  So, I have made a small alteration to your scripts to specify the data sub-directory – e.g.: in prepare_data.py(line 6) the path param for the open call is now for open('data/tweets.js', ....  This way I can keep my(your) scripts separate from the clutter in the top level archive folder.

That is all for now – so far so good!  🙂

[dtwaling]

Um…  Hello!

Yet another propeller-head here, trying to keep up with the speed of technological advancements.

I’m using the monkey pfp for consistency, as it is a common image I’ve used on other forums, like twitter.  You can get a rough idea of who I am via my twitter profile, although my approach is generally much more serious when taking on serious topics and discussions (despite my pfp’s silly monkey appearance).

[Author]

Posts