1685732120ChatGptGeneratedFtpCode
ChatGptGeneratedFtpCode

(This is going to be edited. The code formatted and such.

So I played around with the bing chatgpt instance a little bit and it did a lot of interesting work. But, in the end the mirroring code didn't work.

This shit is getting crazy: ChatGptGeneratedFtpCode



"Write me a python function that will upload a directory tree using ftp"

import os
from ftplib import FTP

def upload_directory_tree(ftp: FTP, local_dir: str, remote_dir: str):
"""
Uploads a local directory tree to a remote FTP server.

:param ftp: An authenticated FTP connection object.
:param local_dir: The path to the local directory to upload.
:param remote_dir: The path to the remote directory to upload to.
"""
for item in os.listdir(local_dir):
local_path = os.path.join(local_dir, item)
remote_path = f"{remote_dir}/{item}"
if os.path.isfile(local_path):
with open(local_path, "rb") as f:
ftp.storbinary(f"STOR {remote_path}", f)
elif os.path.isdir(local_path):
try:
ftp.mkd(remote_path)
except:
pass
upload_directory_tree(ftp, local_path, remote_path)
Copy
This function takes an authenticated FTP connection object, the path to the local directory to upload and the path to the remote directory to upload to. It recursively uploads all files and directories within the local directory tree to the remote FTP server.

Is there anything else you would like me to add or explain?


----

> Write me a python function that will upload only new or changed files from a directory tree using ftp

import ftplib
import os

def upload_new_or_changed_files(ftp_host, ftp_user, ftp_password, local_dir, remote_dir):
ftp = ftplib.FTP(ftp_host)
ftp.login(ftp_user, ftp_password)
for root, dirs, files in os.walk(local_dir):
for file in files:
local_file_path = os.path.join(root, file)
remote_file_path = os.path.join(remote_dir, os.path.relpath(local_file_path, local_dir))
try:
ftp.cwd(os.path.dirname(remote_file_path))
except ftplib.error_perm:
ftp.mkd(os.path.dirname(remote_file_path))
ftp.cwd(os.path.dirname(remote_file_path))
with open(local_file_path, 'rb') as f:
try:
ftp.size(remote_file_path)
print(f"{remote_file_path} already exists. Skipping.")
except ftplib.error_perm:
print(f"Uploading {remote_file_path}")
ftp.storbinary(f"STOR {remote_file_path}", f)
ftp.quit()

----

> How can I modify this function to delete files on the remote server that are not present in the local directory?

import ftplib
import os

def mirror_local_directory(ftp_host, ftp_user, ftp_password, local_dir, remote_dir):
ftp = ftplib.FTP(ftp_host)
ftp.login(ftp_user, ftp_password)

# Get a list of all files on the remote server
remote_files = []
ftp.retrlines('LIST', remote_files.append)
remote_files = [f.split()[-1] for f in remote_files]

# Create directories and upload files
for root, dirs, files in os.walk(local_dir):
for dir in dirs:
local_dir_path = os.path.join(root, dir)
remote_dir_path = os.path.join(remote_dir, os.path.relpath(local_dir_path, local_dir))
try:
ftp.cwd(remote_dir_path)
except ftplib.error_perm:
ftp.mkd(remote_dir_path)
for file in files:
local_file_path = os.path.join(root, file)
remote_file_path = os.path.join(remote_dir, os.path.relpath(local_file_path, local_dir))
try:
ftp.cwd(os.path.dirname(remote_file_path))
except ftplib.error_perm:
ftp.mkd(os.path.dirname(remote_file_path))
ftp.cwd(os.path.dirname(remote_file_path))
with open(local_file_path, 'rb') as f:
try:
ftp.size(remote_file_path)
print(f"{remote_file_path} already exists. Skipping.")
except ftplib.error_perm:
print(f"Uploading {remote_file_path}")
ftp.storbinary(f"STOR {remote_file_path}", f)
if remote_file_path in remote_files:
remote_files.remove(remote_file_path)

# Delete any remaining files on the remote server
for file in remote_files:
print(f"Deleting {file}")
ftp.delete(file)

ftp.quit()

mirror_local_directory('ftp.example.com', 'username', 'password', '/path/to/local/dir', '/path/to/remote/dir')

----





----
PostMe
CategoryProgramming
CategoryAi
CategoryChatGpt

1685732096ChatGptCoding
ChatGptCoding

So over the last couple weeks, as I worked on this site replacement, I experimented with Bing's AI chat feature. Seeing as how Microsoft bought OpenAi, I figure that's backended with ChatGpt.


I did some experiments:

One of the things I have the site code doing is converting from something close to MarkDown into html. Of course the stuff I've got isn't strictly Markdown (I have an awful lot more weird little tags and edge cases that make it nonviable as a 1:1 spec) , but is close enough to use it as a starting point.

First, a couple weeks ago I asked the AI to write a python function that converted Markdown into html, over a couple iterations. The first thing it did was give me a one-liner that used the Markdown module, took a filename and spit back a coverted block of content.

Cute, but completely fucking useless if I'm going to add my own hooks and such.

So I tried again and said "..without using the Markdown module" and it produced a passable function, about 50 lines, that did more or less what I'd asked.

Then a funny thing happened. I put it all down and went about my week, returning to it a week later.

Now, the bing AI interface on windows seemed to require their browser. Fair enough if I'm gonna keep it separate and all. So I restarted Edge, which I hadn't used since that previous attempt.

It started with the old session more or less in tact, so what it did when it loaded the page was resubmit my prompt.

Interestingly I got a function that would parse markdown to html (I mean, duh.) But the algorithm for doing so was completely different. Startlingly different.


One of the other things this code does is generate the entire website, then upload anything that's changed over the top of the existing one.

So I went with another prompt:

"Write a python function that recursively uploads a directory tree using FTP."

And it generated code that looked fine. Interesting, okay. Well, let's play with it a bit.

I let it rip against a directory of about 450 files. It uploaded...most of them, but kept puking on an invalid character error. Odd. I poked around a bit and found that there were files starting with a lowercase 'r'. Any nerds out there figure this out? Note that I'm running from a windows dev box (not that the AI knew that.)

Yep. It built the path using backslashes. That's fine MOST of the time. But if there's an '\r' in a filename on the ftp upload, it puked because the server treated it like a line-feed in the filename. Interestingly FTP is pretty agnostic about the path separators. So there's no reason not to use forward slashes.

"Write a python function that recursively uploads changed files in a directory tree using ftp."

That created something a little different and used comparisons of file sizes as the determinant as to whether any particular file had "changed." Sensible, but not actually accurate. Easy enough to replace that determinant with a "did_file_change" method.

I decided to push my luck.

"Modify that function so that it mirrors the local subdirectory on the server."

The idea was to see if it would be able to add the logic to delete remote files that had been removed from the client side. That would give me full master-slave mirroring, which is what was actually necessary.

Yeah, it couldn't do that.



----
PostMe
CategoryPost
1685732091TwentyTwentyThreeJuneFirst
TwentyTwentyThreeJuneFirst

RabbitRabbit and all.

I'm back on twitter after what seems like six fucking months. I suppose it was the whole "NO social media of any kind" that really caused the time to stretch the hell out like that. But damn if that didn't drag the hell by.

I replaced the site yesterday with the home-grown one from the StaticSiteGenerator. It's off to a good start but there is an absolute metric shitton of work to do from here.

I'm mildly nauseous, which isn't great for productivity.

ChatGptCoding

----
PostMe
TwentyTwentyThreeJune
1685732071WilsonianRantitude
WilsonianRantitude


We all know what this is.

In the immortal words of the late JohnPinette: "Apparently I'm very entertaining when I'm raging."



----
PostMe
1685732024UniversalChurchOfCosmicUncertainty
UniversalChurchOfCosmicUncertainty

The UCCU was my first(ish) blog back on the RadioUserland platform in the early 2000s.

----
PostMe

CategoryBlog
1685731981TwentyTwentyThreeJuneSecond
TwentyTwentyThreeJuneSecond

Well, if you're seeing this then it works. Of course, what IT is you'll have to keep reading to find out.

Okay I wanna get some shit done on the site today. I've only got about 3 hours. I'd HOPED I'd have more time, but I spent the morning dicking around and finally hitting the trigger on that amazon cart I'd been avoiding for a month.


I think the primary focus has to be wiring the thing up to automation.

So that means:

- WikiImporter
- StaticSiteGenerator
- Data push.

Now all of those things work in isolation just fine. But pipelining them could be a bit weird. I REALLY only want subsequent steps to kick off "if and only if" there is new or modified content. Shouldn't be too bad, but it does suggest a top-level wrapper script of some kind. I don't mind, for instance, the StaticSiteGenerator kicking off every 15 minutes, since it runs locally and will just check to see if there's anything to do then do it or not.

But the data push is a different matter. I really don't want that to run every n minutes, log in to the server and do a recursive comparison. This suggests long term that I'm going to need to keep some kind of state of the site sentinel locally so I can at least see if I can afford to shortcut anything. Sure, I'll always be able to force a push. But I don't want that to be my default use case.

There's an awful lot of optimization that needs to happen. But it works for now, so so be it.

I'm going to run some experiments with this post in particular, so there are likely (well...hopefully) going to be a couple iterations of it that change throughout the course of the day.

The real trick is going to be taking the laptop to Smokey, editing it over there and seeing if the changes propagate. They should. But Murphy has a sense of humor with that kind of crap.

And for the most of you who couldn't care less about this crap, yes I'm going to be changing how I organize words on here. There will be a couple/few top-level categories of post types. Things like daily notes, actual expository effluvia, reference posts (my chili recipe, etc.) and probably quick quips. Hell, I may even introduce a "what I'm currently listening to" if I can finagle that kind of thing out of youtube.

One thing that IS going to happen is that a lot of these normal pages from my DesktopWiki are going to start appearing. I'm going to try and hold back the tide until I've got the aforementioned "types of posts" differentiated. No one wants to read the flood of infrastructure pages if they're here for WilsonianRantitude or old UniversalChurchOfCosmicUncertainty stuff.


Designers took the coolness of the web from us in the early 2000s. They made it more complex and less appealing. Time to do something about that.


----
PostMe
TwentyTwentyThreeJune
1685731835StaticSiteGenerator
StaticSiteGenerator

Migrate a bunch of this stuff to the ProjectPlanStaticSiteGenerator page.

I've got the StaticSiteGenerator working on generating a static site out of wiki pages. But now that I've got the WordPressImport pulling those articles into rfc822 files I think what I may do is back off of that code and pull the wiki into the same format as an interim storage medium.

Sure, I could easily enough fold all the extraction code from the WordPressImport and the existing page generator together so it can be run stateless. But that's a nasty nasty design mistake. Better to have discrete tools that run the data through stages.


_ Abstract the configuration information
_ Push to HhcServerPi
_ Add WordPressImport
_ Build SiteTemplate system. Right now the thing just generates pages from this wiki. That's not strictly sufficient. I suppose what I really want, and hadn't thought of it 'til just now is for it to generate two separate sites. One isn't really "static"...okay yeah, that's a separate piece of software that shares components of the rendering engine but uses the live wiki decorations, metadata, linkages and such.

Okay yeah. So that actually makes sense. The DesktopWiki that I've got going, both here in emacs and on the internal web site, is a live server (and also has to be migrated to the HhcServerPi.) But the StaticSiteGenerator is different.

Ugh. This is going to be a bit of a bear.

It's time to start screwing around with the markup language I'm going to be using for extracting and round tripping TodoList stuff, whether that's to TodoIst or not, though I expect it should be.

So actual NextAction items:

_ Move existing WikiServer to new machine.
_ Pull DesktopWiki into the StaticSiteGenerator.
_ build a DeploymentProcess
_ set up a source repo for hhc code
_ set up a repo for the actual pages themselves.
_ WikiTodoExtractor
_ Import UCCU Blog posts




-----
PostMe
CategoryHhc
CategoryProject
1685559377TwentyTwentyThreeMayThirtyFirst
TwentyTwentyThreeMayThirtyFirst


What the cinnamon toast fuck happened here!?!


Welp, today's the day. Yep. I'm making good on my "Get the site replaced by the end of May." I'm pushing this shit over the wall if I'm going to need to stay up 'til 5 in the morning to do it, though...I don't suspect I will.

YES, I know this is super spartan. It's very 1995.

I've been thinking about the problem of preserving links between the old and the new sites. Things like the 9/11 account and such.

But...I think I'm going to not try and do that. It's too much horseshit just for the sake of accomodating the past; something that's becoming a bit of a theme for me lately.

Instead what I've done is snipped off the head of the WordPress site, duplicated most of the posts in here, and am just running this, for a while at least, in parallel. That way the old links will all still work, but that site will be frozen forevermore, with the possible exception of completely disabling the commenting system, if I can figure out how to do that easily.

So what am I really doing here, aside from making DigitalMudPies?

I've replaced WordPress with my own home-brew site generation code which runs against my PersonalKnowledgeManagementSystem, finds things that are marked for posting, and posted them. Of course it does that while having preserved the original website's articles, if not, as I mentioned, links. But I think with a couple exceptions I'm going to be importing all of the original blog posts into the local wiki which is going to have some actual far-reaching implications for how I think and process information.

For one, posts will change over time as they'll be reflecting my own edits and evolution of ideas. Because of that I'm going to start spending more energy honing the structure of the content itself. I'll probably start more stream of consciousness posts, but over time they're going to close in on coherence a bit. Or well, that's the hope anyway.

My plan is to add, over the coming weeks, things like a RecentChanges sidebar that'll differentiate between "new posts" and actually changed content.

Anything YOU see is going to be filtered through a process that flags things for publish. So no, you're not getting the naked, few thousand pages that I have already just dumped out for all to see. There's too much shit in that corpus that's simply NOT for public consumption.

So, when you look down this post you'll see the Category WikiWord and the PostMe tag. When those pages become live, the links themselves will "automatically" turn in to proper links.

So yes, the style is horrendous. The timestamp at the bottom of the article is in unix epoch time. The list of articles on the front page is a fucking html table. There's a LOT to change. But the framework is there. There are actually almost 500 pages in this static version of the site, comprising mostly the original WordPress website. (I was on the fence.)

Over time I'm going to add a metric shitton of "PostMe" tags to existing pages in my wiki repository. Currently this blog, throughout time has accumulated about 1.3 million words. My local PKMS has at least four times that.


Let's see, what else...

OH!

- The rss feed is just...sitting there as it was before I did this cutover.
- I've disabled statcounter, seeing as how I have naked httpd logs.


This is all shit on the list of shit to fix.

One thing that I have NO intention of fixing is that there's no interactivity. Sorry peeps. I've gotten a half dozen comments I value tremendously over the past couple decades. But having WordPress installed leads to thousands of hack attempts and spam comment attempts per DAY, even on this shitty little site. So this is now just a plain old website of static html and over time no dobut a bunch of images. There's nothing to hack. The whole site is the result of a process that generates it on my home server, then pushes the whole thing up to the machine you're currently pointed to, backed up ad nauseum.

I'll come up with some contact form. But it may be a little old-school web email form that uses your client. Hell, if I was feeling clever I could use THAT to automatically add comments. But even that sounds like a spam vector to me so probably not.

Now that the framework is there I can just bang out quick fixes (and not so quick fixes) over time. I don't expect the file layout and link structure to change TOO much over time. But...feh. That's why I left the wordpress site in place.

Feels good. But now I think I'm going to hit publish on all this happy horseshit and go smoke something fucking expensive.



----

PostMe
CategoryTwentyTwentyThreeMay
1685551860ProjectPlanStaticSiteGenerator
ProjectPlanStaticSiteGenerator

It's time to create something of a formal ProjectPlan for the StaticSiteGenerator


Okay I've put a determinant flag in the Page class so I can selectively in/exclude pages from the publishing process. That way I'll be able to do an immediate cut over with the WikiPages embedded.

BUT it's not engaged. The factory for the Wiki page class and the iWilson class circumvents the repository manager.

SO

First tasks:


_ Get rid of that 'more stuff here' sidebar
_ expand the recent changes list (horizontally)
_ Change the page publish template to add: _ time, _ root backlink.

- Disable wordpress.
- Hit the big red button

x Verify the wiki page class's to_publish() method against the 'PostMe' (nee DispositionPublish) 'category' membership.
x Change the page list method so that it locally instantiates the page objects in the repo man object.
x Iterate across those when actually grabbing the recent page list, actual individual pages, or the full page list.





NextTask:


_ Add a next/previous system
_ Translate internal static links (they still point to the wp-upload directory)
_ Create static subdirectory for image uploads, etc.
x Make the uploader recursive.
x Move page footer to article footer
x Change the article page title text to the actual title text instead of the page link fragment.
_ Recent Page module

0.1 MVP:

The 0.1 site will consist of individual pages from the original site, published with their original links BUT in a one-off directory.
*-
x Determine What to publish. Currently using a simple "if it's an iwilson file, publish it."
_ Configure 'root' url.
_ publish to temporary subdirectory (iwilson.net/staging/)
_ Stage pages
_ Manually ftp


Full list of goals:

_ Expand the "what to post" filter to include the 'posted' flag from wordpress (or maybe...change the wordpress importer to translate that into a disposition flag that'll match the one from the wiki import.)
_ rss and, sure wtf, atom feeds
_ Move existing WikiServer to new machine.
x Pull DesktopWiki into the StaticSiteGenerator.
_ build a DeploymentProcess
_ set up a source repo for hhc code
_ set up a repo for the actual pages themselves.
_ WikiTodoExtractor
_ Import UCCU Blog posts
_ Some of the links on the wordpress import are fucky. They're pointing to 'directory' entries rather than pages, notably the index post links.
_ wordpress importer: quote import.
_ wordpress importer: Comma import.
_ wordpress importer: Youtube links need to be converted to...I don't even know what. Some custom pseudo-markdown 'embed' thing? gak.



----
CategoryProjectPlan
CategoryStaticSiteGenerator
CategoryBlog
CategoryWiki