diff options
| author | s-ol <s-ol@users.noreply.github.com> | 2020-02-15 14:38:17 +0000 |
|---|---|---|
| committer | s-ol <s-ol@users.noreply.github.com> | 2020-02-15 14:38:17 +0000 |
| commit | 3abb5c0be1bba44702a77a32caf368775ab855c2 (patch) | |
| tree | 5451aae8f5eb499094fd3b6f0338150798f8adf9 /root/articles/mmmfs/ba_log | |
| parent | fix conversion want wildcard (diff) | |
| download | mmm-3abb5c0be1bba44702a77a32caf368775ab855c2.tar.gz mmm-3abb5c0be1bba44702a77a32caf368775ab855c2.zip | |
rename articles to research
Diffstat (limited to 'root/articles/mmmfs/ba_log')
27 files changed, 0 insertions, 791 deletions
diff --git a/root/articles/mmmfs/ba_log/$order b/root/articles/mmmfs/ba_log/$order deleted file mode 100644 index a77277a..0000000 --- a/root/articles/mmmfs/ba_log/$order +++ /dev/null @@ -1,15 +0,0 @@ -start -2019-10-07 -2019-10-08 -2019-10-09 -2019-10-10 -2019-10-11 -2019-10-14 -2019-10-15 -2019-10-24 -2019-10-26 -2019-10-27 -2019-10-29 -2019-11-01 -2019-11-25 -2019-12-20 diff --git a/root/articles/mmmfs/ba_log/2019-10-07/text$markdown.md b/root/articles/mmmfs/ba_log/2019-10-07/text$markdown.md deleted file mode 100644 index 46c6892..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-07/text$markdown.md +++ /dev/null @@ -1,40 +0,0 @@ -Today I started working on the HTTP server that finds, converts and serves content stored in the (SQL) backend just-in-time (later the server could also cache content). - -The server can handle these types of requests: - -## Fileder Index Requests -A request like `GET /path/to/fileder/` (note the trailing slash) is used to query the contents of a fileder. -It solicits a JSON-encoded response that contains the full paths to all children of this fileder, as well as all facets currently stored, e.g: - - { - "children": [ - "/projects/vcv_mods", - "/projects/HowDoIOS", - "/projects/iii-telefoni", - "/projects/btrktrl", - "/projects/demoloops", - "/projects/VJmidiKit", - "/projects/gayngine", - "/projects/themer", - "/projects/chimpanzee_bukkaque" - ], - "facets": [ - ["", "text/moonscript -> fn -> mmm/dom"], - ["name", "alpha"], - ["title", "text/plain"] - ] - } - -## Facet Requests -A request like `GET /path/to/fileder/facet_name` is used to query a facet. -To differentiate a request for the 'unnamed' facet from an index request, unnamed facets are represented as a `:` character instead. -The type to ask for can be specified in a `MMM-Accept` header separately, it defaults to `text/html`. - -The server either sends back the (possibly converted) facet with a `200 OK` status, -or a `406 Not Acceptable` error if no conversion was possible. - -I also restructured the code a bit and moved some of the HTML-rendering code into the main mmmfs code. -Then I renamed the `text/html` type to `text/html+frag`, since it refers to only a fragment of HTML code, not a whole document, -and added a new *convert* from `text/html+frag` to `text/html` that wraps the fragment in the HTML template and style. - -the full code change is in commits [81e143f](https://git.s-ol.nu/mmm/commit/81e143fa8181a6adb58d7fba632bd31a13164410/) and [ad26c7c](https://git.s-ol.nu/mmm/commit/ad26c7c4e374f66a978f9946bbb083377f2224a6/) diff --git a/root/articles/mmmfs/ba_log/2019-10-08/text$markdown.md b/root/articles/mmmfs/ba_log/2019-10-08/text$markdown.md deleted file mode 100644 index c10b75f..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-08/text$markdown.md +++ /dev/null @@ -1,58 +0,0 @@ -Today I mostly fixed the output/rendering of the 'live' server I implemented yesterday. - -I changed the URL scheme, it no longer uses headers, which made it hard to link to resources through `<link>` and `<script>` tags. -Instead the path, facet-name and type are now all part of the URI: - -| URI | fileder path | facet name | type | -| ---------------------------------------------- | ------------------ | --------------- | ------------------------------ | -| `/fileder/facet: type/subtype -> wrapped/type` | `/path/to/fileder` | `facet` | `type/subtype -> wrapped/type` | -| `/fileder/: some/type` | `/fileder` | (default facet) | `some/type` | -| `/fileder/alternate:` | `/fileder` | `alternate` | `text/html` (default type) | -| `/path` | `/path` | (default facet) | `text/html` (default type) | -| `/` | `/` | (default facet) | `text/html` (default type) | - -The fileder-index metadata was moved to a `?index` 'pseudofacet' (e.g. `/fileder/?index`). - -**EDIT 2019-10-09:** -> I will take this chance to show and hopefully explain the point of this all again: -> With the system running it is now possible to demonstrate the type-coercion that powers the system. -> -> In the system, every piece of data (the *facet*s) is stored together with it's *type*. -> When requesting data, it can of course be loaded with that *type*, yielding the data unmodified, -> but it is also possible to demand a different *type* that may be more useful to the receiving application. -> -> as an example we can take for example this article about one of the internal libraries for writing HTML documents: -> [`/meta/mmm.dom/`](/meta/mmm.dom/). -> -> The path `/meta/mmm.dom/` corresponds, according to the table above, to the default facet of the `/meta/mmm.dom` fileder, -> and since no type was specified, it is assumed that the browser wants a `text/html` document. -> The follwing path gives the same result, but makes this explicit: -> [`/meta/mmm.dom/:text/html`](/meta/mmm.dom/:text/html) -> -> now, instead of asking for the rendered HTML document, we can also ask for the source, which is of the type -> `text/moonscript -> mmm/dom` (a Moonscript file that evaluates to a website-fragment): -> [`meta/mmm.dom/:text/moonscript -> mmm/dom`](/meta/mmm.dom/:text/moonscript%20-%3E%20mmm/dom) -> -> the source code was written in MoonScript, which compiles to Lua before it is executed. -> We can also ask the server to do that for us: -> `text/lua -> mmm/dom` (a Lua file that evaluates to a website-fragment): -> [`meta/mmm.dom/:text/lua -> mmm/dom`](/meta/mmm.dom/:text/lua%20-%3E%20mmm/dom) -> -> or, we could ask for the generated html fragment, but without the full HTML layout around it - -> that would be the type `text/html+frag`, as mentioned in the last post: -> [`meta/mmm.dom/:text/html+frag`](/meta/mmm.dom/:text/html+frag) -> -> lastly, we could also ask the system to generate a link to this content, in the `text/html` format (`URL -> text/html`): -> [`meta/mmm.dom/:URL -> text/html`](/meta/mmm.dom/:URL%20-%3E%20text/html) -> this might seem somewhat redundant, since we need a link to access this link, -> but it can be useful when a component cannot work with binary data directly, e.g. when mentioning or embedding an image -> or a video file. - -I also added support to the server for serving static assets (e.g. the CSS stylesheet) from the `static` directory. -These files are accessible through the `/.static/` route (e.g. `/.static/main.css`), where they shouldn't interfere -with the mmmfs contents. -With the layout adjsuted to use these new paths, the live server now looks properly styled again too! - -Finally I worked on the Dockerfile and my deployment a bit, -so that my updates to the code will now be automatically applied to my test site, -which is available at [ba.s-ol.nu](//ba.s-ol.nu) (which might be where you are reading this right now!). diff --git a/root/articles/mmmfs/ba_log/2019-10-09/text$markdown.md b/root/articles/mmmfs/ba_log/2019-10-09/text$markdown.md deleted file mode 100644 index 702620d..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-09/text$markdown.md +++ /dev/null @@ -1,9 +0,0 @@ -I added another driver/store that loads files straight from disk \[[`86bbe80`][86bbe80]\], -and made the server load the fileder tree when it receives a request for content, rather than loading the whole tree up front. -This means that I can work on the content again and see changes in the browser without restarting the server every time \[[`97bc4a0`][97bc4a0]\], -This feature should be made unnecessary by the in-page editing feature, but until then it's important for my workflow. - -I also started cleaning up the mmmfs article a bit, and integrating this project log in a way that will make it available online soon. - -[86bbe80]: https://git.s-ol.nu/mmm/commit/86bbe805a7ec49a8b891412713ea43d6e46d0d73/ -[97bc4a0]: https://git.s-ol.nu/mmm/commit/97bc4a0d8d866026905eac6f0ba08b75f166219a/ diff --git a/root/articles/mmmfs/ba_log/2019-10-10/text$markdown.md b/root/articles/mmmfs/ba_log/2019-10-10/text$markdown.md deleted file mode 100644 index 010f861..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-10/text$markdown.md +++ /dev/null @@ -1,46 +0,0 @@ -Today I moved the static resources needed by the web-frontend into the content of the mmmfs itself: -The server doesn't make an exception for static files anymore (as described in [a previous update][2019-10-08]), -but rather the files are just in a fileder called `static` now, and properly typed, here: - -- [/static/style/](/static/style/:%20text/html+interactive) -- [/static/mmm/](/static/mmm/:%20text/html+interactive) - -This removed a big exception and left the server implementation much cleaner and shorter, as can be seen in the corresponding commit [`005cc9b`][005cc9b]. - -I also changed the route syntax introduced in [`2019-10-08`][2019-10-08] for getting the fileder index, -now instead of being hard-coded to return a JSON value at `?index` \[[`b36a1a6`][b36a1a6]\]. -`?index` is treated as a pseudo-facet that can be requested in different types, just like real facets. -I also added a second pseudo-facet `?tree`, which works like `?index`, except that it recurses and includes -all content below the current fileder, rather than just including the child-fileders. - -Here are some example links for viewing these: - -- [`/?index: text/html`](/?index:%20text/html) -- [`/articles/mmmfs/ba_log/?index: text/html`](/articles/mmmfs/ba_log/?index:%20text/html) - -Finally I added a third pseudo-facet called `?interactive` that renders the Inspector that the old page ran on, -allowing to inspect raw facets, and bringing back the navbar \[[`9ab2f0f`][9ab2f0f]\]. - -Now that there was a way to serve the Browser to the client again, I got to work on fixing it there. -This involved a bigger changes in the shared mmm internals: - -Up to this point, for each request to render a fileder, the server would load that fileder, -with all its facets and their values, as well as all children and their facets and children recursively. -That means that when the root fileder is rendered, currently 120MB of data have to be loaded from disk (or a database). -Now that the client can render content within the web browser again, that would be even worse due to the network delay. - -To solve this, the Fileder implementation now lazy-loads \[[`9632233`][9632233]\]. -When a Fileder is created, it initially knows only its path, but doesn't know which facets or children it contains. -As soon as that data is attempted to be accessed, the fileder loads in the list of its children and facets from the datastore. -The facet contents are loaded only when they are actually needed to fulfill a data request or conversion. - -With this need for optimization taken care of, I added a new datastore (`web`) \[[`91546d1`][91546d1]\], that can run on the client. -Instead of directly accessing a database or physical file system, like the `sql` and `fs` stores, the `web` store delegates all -requests to the server APIs I have been building, such as the new `?index` pseudo-facet. - -[2019-10-08]: /articles/mmmfs/ba_log/2019-10-08/ -[005cc9b]: https://git.s-ol.nu/mmm/commit/005cc9b3914128267017620984aee921999e173f/ -[b36a1a6]: https://git.s-ol.nu/mmm/commit/b36a1a6c61a6e8bff156ce4e2dc66fe8ed8cd95e/ -[9ab2f0f]: https://git.s-ol.nu/mmm/commit/9ab2f0fe3a1a043300536a057bafe5058d987d7f/ -[9632233]: https://git.s-ol.nu/mmm/commit/9632233c16a26f017c648faf36a6b26833e62f2e/ -[91546d1]: https://git.s-ol.nu/mmm/commit/91546d12919736b08567d7174bf1063cab0838f0/ diff --git a/root/articles/mmmfs/ba_log/2019-10-11/text$markdown.md b/root/articles/mmmfs/ba_log/2019-10-11/text$markdown.md deleted file mode 100644 index 0b58c75..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-11/text$markdown.md +++ /dev/null @@ -1,30 +0,0 @@ -Yesterday I got client-side access to the whole mmmfs tree to work via the new `web` datastore, -but the requests were still extremely inefficient. -This was because the organisation of the interface between the `Fileder` implementation and the datastore, -which required the Fileder to make two separate requests to `?index` to fetch its children and facets, -and also couldn't take advantage of the `?tree` pseudo-facet to bundle together multiple indexes in a single request. - -To fix this I added a new datastore method called `get_index` that returns both facet and child information \[[`e2a4257`][e2a4257]\] -The Fileder implementation now uses this method instead of asking individually for the two pieces of information -when lazy-loading fileder contents. -The `get_index` method can also be instructed to recursively load to a fixed depth. -A Fileder can also be instantiated using such a nested index, -which causes it to immediately preload up to the same depth without the need for fetching more data. -This allows some more optimizations, like having the client preload 3-levels deep when it launches, -which seems like a decent heuristic of the data actually required for most pages and minimizes load time. - -I also added some tests for the `Key` class that I use in many different places to represent facet names and types \[[`782d072`][782d072]\]]. - -The `?interactive` pseudo-facet from yesterday was changed to the `text/html+interactive` type instead \[[`8cdf5d4`][8cdf5d4]\]. -This was a bit of a tough decision, because it is a bit un-idiomatic: rendering the browser page still requires an exception in the browser. -In the end the motivation for the change was that it should be possible, -for user ergonomics, to link to the interactive view of a given facet of a given fileder. -With `?interactive` being a facet, it wasn't possible to specify the facet without jumping out of the adressing system. -With these updates on the other hand it is possible to link fo example: - -- to the main content of the root fileder, as an interactive view: [`/: text/html+interactive`](/:%20text/html+interactive) -- to the page title of the root fileder, as an interactive view: [`/title: text/html+interactive`](/title:%20text/html+interactive) - -[e2a4257]: https://git.s-ol.nu/mmm/commit/e2a4257fc05d37822df2b7bbe0f587645375edf2/ -[782d072]: https://git.s-ol.nu/mmm/commit/782d0725f3f29eaa7d4a12213fb00c6643795348/ -[8cdf5d4]: https://git.s-ol.nu/mmm/commit/8cdf5d4a363ba99a6356e7e1dfe0dfb39e6fb13e/ diff --git a/root/articles/mmmfs/ba_log/2019-10-14/text$markdown.md b/root/articles/mmmfs/ba_log/2019-10-14/text$markdown.md deleted file mode 100644 index e42f8be..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-14/text$markdown.md +++ /dev/null @@ -1,101 +0,0 @@ -I finally added a better type-conversion-path-finding algorithm \[[`deb21aa`][deb21aa]\]. -The old algorithm only took into consideration how many steps it took to get from type A (stored on disk) to type B (requested). -This could lead to problems, for example in this situation: - -On Disk: `text/moonscript -> fn -> mmm/dom` (a moonscript file that contains a function that returns a bit of UI) -Requested: `mmm/dom` -Known Conversions: -1. `text/moonscript` to `mmm/dom` (displays the source code with highlighting) -2. `text/moonscript -> ???` to `???` (evaluates the moonscript file) -3. `fn -> ???` to `???` (calls the function, providing access to children and other facets) - -Since conversion 1 only takes a single step, it would have been preferred by the old algorithm (although there were workarounds for this). -The new algorithm adds the concept of conversion-cost, that has to be specified for each conversion. -The conversions 2 and 3 now have a cost of `1`, while conversion 1 has a cost of `5`. -The conversions are simply added up and the path with the lowest cost is chosen. -Like in other pathfinding applications like digital games, the cost metric is also used by the algorithm to enhance the search itself, -it prioritises searching further on the path with the least current cost. - -To implement this optimization I implemented a priority queue: - - -- a priority queue with an index - -- only one element with a given key may exist at a time - -- when an element with an existing key is added, - -- the element with lower priority survives. - class Queue - new: => - @values = {} - @index = {} - - -- add a value with a given priority to the queue - -- if no key is specified, assume the element is uniq - add: (value, priority, key) => - entry = { :value, :key, :priority } - - if key - if old_entry = @index[key] - -- already have an entry for this key - -- if it is lower priority, we leave it there and do nothing - if old_entry.priority < priority - return - - -- otherwise we remove the old one and continue as normal - -- find the index of the old entry - local i - for ii, entry in ipairs @values - if entry == old_entry - i = ii - break - - -- remove it - table.remove @values, i - - -- store this entry in the index - @index[key] = entry - - -- store lowest priority last - for i, v in ipairs @values - if v.priority < priority - -- i is the first key that is lower, - -- we want to insert right before it - table.insert @values, i, entry - return - - -- couldn't find a key with a lower priority, - -- so insert at the end - table.insert @values, entry - - peek: => - entry = @values[#@values] - if entry - { :value, :priority, :key } = entry - @index[key] = nil if key - value, priority - - pop: => - entry = table.remove @values - if entry - { :value, :priority, :key } = entry - @index[key] = nil if key - value, priority - - -- iterator, yields (value, priority), low priority first - poll: => @.pop, @ - - { - :Queue - } - -This priority queue behaves mostly as expected, but I added an extra feature to make sure that only the -best conversion path to a specific type is considered for further searching: -When adding a new element to the Queue, an extra `key` can be passed. -If a key is passed, the Queue makes sure that there is only ever one element with that key in the queue. -When a second element would be introduced, the queue discards whichever element has a higher priority value. - -This way I can use the type that a conversion path leads to as the key, -and the queue will automatically discard a worse solution if a better one is found that leads to the same type result. - -To make sure the Queue implementation was solid, I also added unit tests for it: [`spec/queue_spec.moon`][spec] - -[deb21aa]: https://git.s-ol.nu/mmm/commit/deb21aa43fe8bf11eb276803973b272913b7e716/ -[spec]: https://git.s-ol.nu/mmm/blob/deb21aa43fe8bf11eb276803973b272913b7e716/spec/queue_spec.moon diff --git a/root/articles/mmmfs/ba_log/2019-10-15/text$markdown.md b/root/articles/mmmfs/ba_log/2019-10-15/text$markdown.md deleted file mode 100644 index d4e4121..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-15/text$markdown.md +++ /dev/null @@ -1,70 +0,0 @@ -I'm giving a workshop next weekend, and I had to create the slides for that. -Since one of the aspirations of `mmmfs` is to be easily adaptable to any kind of data organization and -presentation task, -that was a good opportunity to try and implement a simple slideshow system. - -I started by creating a new fileder to hold the slideshow, [`/articles/xy-workshop`](/articles/xy-workshop/). -In the main facet, I created a MoonScript file. -Since the facet needs to access its children (the individual slides), I also used the `fn ->` type, -that injects the current fileder into the script. - -`text/moonscript -> fn -> mmm/dom` - - import ReactiveVar, tohtml, fromhtml, text, elements from require 'mmm.component' - import article, button, div, span from elements - - => - index = ReactiveVar 1 - slide = index\map (index) -> @children[index] - - local view - view = div { - style: ... -- styling ommited here - - div { - style: ... -- styling omitted here - - slide\map => @get 'mmm/dom' - } - } - - local left, right - if MODE == 'CLIENT' - left = (_, e) -> - e\preventDefault! - index\transform (a) -> math.max 1, a - 1 - - right = (_, e) -> - e\preventDefault! - index\transform (a) -> math.min #@children, a + 1 - - tohtml with article! - \append div { - button '<', onclick: left - ' ' - span index\map (t) -> text t - ' ' - button '>', onclick: right - } - \append view - -I used the `mmm.component` library that lets me create reactive UIs. -First I declare the reactive variable `index`, the current slide number. -Then I derive the `slide` reactive variable, that is defined to be the child with index `index` - -since these are `ReactiveVar`s, whenever `index` changes, `slide` will automatically load the current slide fileder. - -Then I create the main slide view, which consists mainly of two containers and some styling. -Inside, I derive another reactivevar from `slide`: whenever a new `slide` is selected, -this piece of code will request the main content in `mmm/dom` format and replace the current view content with that. - -Lastly I construct a little navigation UI, consisting of the left and right buttons. -When one of them is clicked, it modifies the `index` variable, making sure to stay in the range of existing slides. -The rest of the UI then reactively updates accordingly. - -Lastly I added keyboard controls for cycling through the slides, as well as a button to enter fullscreen mode. -You can find the extended code for that in the commit [`ca24ef1`][ca24ef1], -but it is not a lot either: with the additions the file is 70 lines long. - -With this done, it is just a matter of creating children-fileders and placing whichever content I want in them, -they behave just like any other page of the system now. - -[ca24ef1]: https://git.s-ol.nu/mmm/blob/ca24ef108dbb11860e719711e4e7fbd6323aee0e/root/articles/xy-workshop/text%24moonscript%20-%3E%20fn%20-%3E%20mmm%24dom.moon diff --git a/root/articles/mmmfs/ba_log/2019-10-24/text$markdown.md b/root/articles/mmmfs/ba_log/2019-10-24/text$markdown.md deleted file mode 100644 index dbb838a..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-24/text$markdown.md +++ /dev/null @@ -1,32 +0,0 @@ -While writing some of the main section of the thesis today, -I felt the need to illustrate some subject matter with a diagram depicting a folder structure. -Knowing of some similar tools that generate SVG diagrams from textual descriptions, -some quick research turned up [mermaid JS][mermaid]. - -I tried it in their online live editor first to verify that it would indeed do what I needed it to, -and then added it to the set of supported types in mmm (on the clientside only) \[[`2ff6f90`][2ff6f90]\]. -This was pretty easy in the end, all it took was adding mermaid.js to the interactive version's resources (in `build/server.moon`) -and then definig the `convert` (and thereby implicitly defining the corresponding type, `text/mermaid-graph`): - - { - inp: 'text/mermaid-graph' - out: 'mmm/dom' - cost: 1 - transform: (source) => - with container = document\createElement 'div' - cb = (svg, two) => - .innerHTML = svg - window.mermaid\render "mermaid-#{id_counter}", source, cb - } - -The code is quite short, since all it needs to do is create a container element, -then tell mermaid.js to render the textual definition. -Once mermaid.js is done, the rendered content is added in the container element. - -Here is one of the two diagrams that I implemented the feature for, -you can also follow the link through to the source and view the textual representation using the 'inspect' button. - -<mmm-embed path="/articles/mmmfs/mmmfs/tree_mainstream"></mmm-embed> - -[mermaid]: https://mermaidjs.github.io/mermaid-live-editor/ -[2ff6f90]: https://git.s-ol.nu/mmm/commit/2ff6f906c498c1b742dd8437a09c97ebe29a652a/ diff --git a/root/articles/mmmfs/ba_log/2019-10-26/text$markdown.md b/root/articles/mmmfs/ba_log/2019-10-26/text$markdown.md deleted file mode 100644 index d5cd9bd..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-26/text$markdown.md +++ /dev/null @@ -1,62 +0,0 @@ -Besides some smaller fixes with the styling of the page, and in particular the diagrams introduced in [`2019-10-24`][2019-10-24], -I finally (re-)implemented children-ordering in the `fs`-store of mmmmfs -(the `sql` store is still missing it, but I am not currently using it either) \[[`a62f63b`][a62f63b]\]: -Files on the regular filesystem don't have a particular order, but in mmmfs the order of children is is guaranteed, -so that arranging children in a particular order becomes a meaningful tool. - -To store the ordering data, a 'magic' file called `$order` is (optionally) stored in each directory in the filesystem. -The file lists all child fileders by name in the given order. -When the children of a fileder are requested (in `list_fileders_in` or `get_index`, which relies on the former), -all children that are mentioned in `$order` are returned in that order, -while all remaining children are sorted alphabetically and appended at the end of the list. -This way the order is guaranteed to be stable even if no `$order` file is specified, -or when the `$order` file has not been updated after adding new children. - -Here is the commented implementation in MoonScript: - - list_fileders_in: (path='') => - -- create a mapping of all child-fileders - -- in 'entries' (name -> path) - entries = {} - for entry_name in lfs.dir @root .. path - continue if '.' == entry_name\sub 1, 1 - entry_path = @root .. "#{path}/#{entry_name}" - if 'directory' ~= lfs.attributes entry_path, 'mode' - continue - - entries[entry_name] = "#{path}/#{entry_name}" - - -- where we will store our sorted list of children - sorted = {} - - -- check for existance of the order file - order_file = @root .. "#{path}/$order" - if 'file' == lfs.attributes order_file, 'mode' - for line in io.lines order_file - path = assert entries[line], "entry in $order but not on disk: #{line}" - - -- add all $order-entries to the sorted output in the same order they appear. - -- also flag these entries as already added - table.insert sorted, path - sorted[line] = true - - -- find the he remaining (non-flagged) entries, sort them alphabetically - -- and then append them to the sorted output list - entries = [path for entry, path in pairs entries when not sorted[entry]] - table.sort entries - for path in *entries - table.insert sorted, path - - -- return an iterator over the sorted output - coroutine.wrap -> - for path in *sorted - coroutine.yield path - -The interface for reordering fileders is still missing in the code, -and just while writing this I realized that the current implementation is in fact buggy: -when a fileder that is mentioned in `$order` is deleted via the `stores.fs` API, -it is not removed from `$order`, causing an error the next time the fileder is listed. -I will probably get around to fixing both of these problems when I build the corresponding UI. - -[2019-10-24]: /articles/mmmfs/ba_log/2019-10-24/ -[a62f63b]: https://git.s-ol.nu/mmm/commit/a62f63bc00cd63a98b349a2574e3e9e14c95a441/ diff --git a/root/articles/mmmfs/ba_log/2019-10-27/$order b/root/articles/mmmfs/ba_log/2019-10-27/$order deleted file mode 100644 index 2b68b52..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-27/$order +++ /dev/null @@ -1 +0,0 @@ -video diff --git a/root/articles/mmmfs/ba_log/2019-10-27/text$markdown.md b/root/articles/mmmfs/ba_log/2019-10-27/text$markdown.md deleted file mode 100644 index 3e9f886..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-27/text$markdown.md +++ /dev/null @@ -1,77 +0,0 @@ -Today I finally reached the ability of editing content inside of mmmfs directly! - -<mmm-embed path="video" nolink></mmm-embed> - -Content is currently not saved anywhere, but the basis of the feature is there. -For the moment this is implemented as yeat another *convert* that simply converts -from `text/.*` (any textual code) to `mmm/dom` (web UI) \[[`28653c9`][28653c9]\]. -However this *convert* is only applied in the inspector view (more on that below). -First, here is the convert itself: - - { - inp: 'text/([^ ]*).*' - out: 'mmm/dom' - cost: 0 - transform: (value, fileder, key) => - mode = @from\match @convert.inp - Editor value, mode, fileder, key - } - -and the main part of the code, the `Editor` widget: - -```moonscript -class Editor - new: (value, mode, @fileder, @key) => - @node = div class: 'editor' - -- 'o' is a little helper for converting a Lua table to a JS object - @cm = window\CodeMirror @node, o { - :value - :mode - lineNumber: true - lineWrapping: true - autoRefresh: true - theme: 'hybrid' - } - - @cm\on 'changes', (_, mirr) -> - window\clearTimeout @timeout if @timeout - @timeout = window\setTimeout (-> @change!), 300 - - change: => - @timeout = nil - doc = @cm\getDoc! - if @lastState and doc\isClean @lastState - -- no changes since last event - return - - @lastState = doc\changeGeneration true - value = doc\getValue! - - @fileder.facets[@key] = value - BROWSER\refresh! -``` - -I chose the [CodeMirror][codemirror] library as the basis for the editor, -because it seemed like one of the leanest ones I could find -(and yet it is quite heavy at 100kb plus styling and language support addons). -The code for the editor is also quite minimale it really just creates a wrapper for the editor -and tells CodeMirror to set itself up inside. -Whenever changes are detected, a timeout of 300ms is started, -after which the preview is refreshed to preview the changes. -If more changes are made within the 300ms, the timer is reset to 300ms, -so that the preview update doesn't interrupt the typing flow. - -I also spent some time refactoring the global list of converts out into multiple smaller plugins -(although a lot of global converts remain at the moment) \[[`abefbf8`][abefbf8]\]. -A plugin can export a list of converts as well as a list of 'editors', which are essentially no different, -except that they are taken into consideration only when converting content for the inspector. - -Breaking the converts up into little packages like that makes it a lot easier to edit, -and allows enabling and disabling individual features very easily. -I am also considering moving the converts into the mmmfs data itself, -to make extending the system and working inside the system more congruent, -and this is a good way of testing my idea of how the modularization of the system should work. - -[codemirror]: https://codemirror.net/ -[28653c9]: https://git.s-ol.nu/mmm/commit/28653c9ae46b2b3e42c2c75879589138c731f37b/ -[abefbf8]: https://git.s-ol.nu/mmm/commit/abefbf82531021f5ca4149675932a7fe2ff37dde/ diff --git a/root/articles/mmmfs/ba_log/2019-10-27/video/video$webm.webm b/root/articles/mmmfs/ba_log/2019-10-27/video/video$webm.webm Binary files differdeleted file mode 100644 index 020e352..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-27/video/video$webm.webm +++ /dev/null diff --git a/root/articles/mmmfs/ba_log/2019-10-29/$order b/root/articles/mmmfs/ba_log/2019-10-29/$order deleted file mode 100644 index 2b68b52..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-29/$order +++ /dev/null @@ -1 +0,0 @@ -video diff --git a/root/articles/mmmfs/ba_log/2019-10-29/text$markdown.md b/root/articles/mmmfs/ba_log/2019-10-29/text$markdown.md deleted file mode 100644 index 4119d21..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-29/text$markdown.md +++ /dev/null @@ -1,38 +0,0 @@ -Today I implemented updating/saving content in the server, and bridged the feature to the client. -In the inspector there is now a `save changes` button that (attempts to) save the content on -the server's filesystem \[[`4b8d9be`][4b8d9be]\]. - -<mmm-embed path="video" nolink>demonstration of editing and persistantly saving facet</mmm-embed> - -Originally I wanted to create a `sandbox` fileder that was to be edited by any one online \[[`d9eafa2`][d9eafa2]\]. -I restricted editing to only fileders underneath `/sandbox`, but then upon publishing quickly realized -that this left open a major security vulnerability, since content can be evaluated on server or client: -if a client were to create a facet `exploit: text/lua -> text/plain` with the following content in the root: - -```lua -pass = io.open('/etc/passwd', 'r') -return pass:read("*all") -``` - -...and then request that facet as converted to `text/plain` (`GET /exploit: text/plain`), -then that Lua code would be executed on the server, and return the confidential `passwd` file on the server. -This basically meant handing anyone online full unconditionaly access to my server -(or at least the VM running the website, and potentially options to escalate from there). - -As a result I had to choose to either disable public editing, or disable server-side code execution. -Because server-side execution is a major feature of mmmfs, I settled for the following compromise \[[`1e3b0a1`][1e3b0a1]\]: - -- when developing and running locally, editing and code execution are both enabled in 'unsafe mode' -- on https://ba.s-ol.nu, editing is disabled but code execution is possible -- on https://sandbox.s-ol.nu, editing is enabled but code server-side code execution is disabled - -The Sandbox can now be found at the following address, at least until the thesis project is concluded: - -# [`sandbox.s-ol.nu`](https://sandbox.s-ol.nu) - -Currently it is only possible to edit existing facets, -but creation and deletion of facets and fileders should be implemented soon. - -[4b8d9be]: https://git.s-ol.nu/mmm/commit/4b8d9be10e4517114c0c216fa24aaaa310503d4a/ -[d9eafa2]: https://git.s-ol.nu/mmm/commit/d9eafa21ad1d6ad340b5744d69e9fc68ce2083cb/ -[1e3b0a1]: https://git.s-ol.nu/mmm/commit/1e3b0a12060dce916b686921c94520202c4cb130/ diff --git a/root/articles/mmmfs/ba_log/2019-10-29/video/video$mp4.mp4 b/root/articles/mmmfs/ba_log/2019-10-29/video/video$mp4.mp4 Binary files differdeleted file mode 100644 index 162d0fc..0000000 --- a/root/articles/mmmfs/ba_log/2019-10-29/video/video$mp4.mp4 +++ /dev/null diff --git a/root/articles/mmmfs/ba_log/2019-11-01/$order b/root/articles/mmmfs/ba_log/2019-11-01/$order deleted file mode 100644 index 1549b67..0000000 --- a/root/articles/mmmfs/ba_log/2019-11-01/$order +++ /dev/null @@ -1 +0,0 @@ -demo diff --git a/root/articles/mmmfs/ba_log/2019-11-01/demo/video$webm.webm b/root/articles/mmmfs/ba_log/2019-11-01/demo/video$webm.webm Binary files differdeleted file mode 100644 index 1463897..0000000 --- a/root/articles/mmmfs/ba_log/2019-11-01/demo/video$webm.webm +++ /dev/null diff --git a/root/articles/mmmfs/ba_log/2019-11-01/text$markdown.md b/root/articles/mmmfs/ba_log/2019-11-01/text$markdown.md deleted file mode 100644 index afcaf0a..0000000 --- a/root/articles/mmmfs/ba_log/2019-11-01/text$markdown.md +++ /dev/null @@ -1,18 +0,0 @@ -In the last two days I implemented the remaining features for full editing support in the browser: - -- support for adding, removing facets in the inspector \[[`dc08e26`][dc08e26]\] -- support for adding, removing fileders in the inspector \[[`ff9ee8e`][ff9ee8e]\] -- implemented the API for reordering fileders as mentioned in [`2019-10-26`][2019-10-26] \[[`053e607`][053e607]\] -- support for reordering fileders in the inspector \[[`16232a2`][16232a2]\] - -Here is a short demo video of these features: - -<mmm-embed nolink path="demo"></mmm-embed> - -As usual, this can be tried out immediately at [`sandbox.s-ol.nu`](//sandbox.s-ol.nu) - -[2019-10-26]: /articles/mmmfs/ba_log/2019-10-26/ -[dc08e26]: https://git.s-ol.nu/mmm/commit/dc08e262cd53a48480a88235aa58500f0638ad79/ -[ff9ee8e]: https://git.s-ol.nu/mmm/commit/ff9ee8e99cd5f5c420ba0501c335ac18f1b10769/ -[053e607]: https://git.s-ol.nu/mmm/commit/053e607a49989b2d4491c20ff14c839b7161d713/ -[16232a2]: https://git.s-ol.nu/mmm/commit/16232a2509a87a900b69b2d0f826a2e3edec3f96/ diff --git a/root/articles/mmmfs/ba_log/2019-11-25/$order b/root/articles/mmmfs/ba_log/2019-11-25/$order deleted file mode 100644 index 1549b67..0000000 --- a/root/articles/mmmfs/ba_log/2019-11-25/$order +++ /dev/null @@ -1 +0,0 @@ -demo diff --git a/root/articles/mmmfs/ba_log/2019-11-25/demo/video$webm.webm b/root/articles/mmmfs/ba_log/2019-11-25/demo/video$webm.webm Binary files differdeleted file mode 100644 index bdf9d9c..0000000 --- a/root/articles/mmmfs/ba_log/2019-11-25/demo/video$webm.webm +++ /dev/null diff --git a/root/articles/mmmfs/ba_log/2019-11-25/text$markdown.md b/root/articles/mmmfs/ba_log/2019-11-25/text$markdown.md deleted file mode 100644 index 48a0bb4..0000000 --- a/root/articles/mmmfs/ba_log/2019-11-25/text$markdown.md +++ /dev/null @@ -1,89 +0,0 @@ -There was a longer break in development of the projects as I have been focusing on the thesis, -where progress is not represented accurately in the repository. - -There was also some progress on featurse that haven't been tidied up and committed yet, -such as drag'n'drop / direct file upload. Those features will probably get their own post sometime soon. - -Today I spent some time to implement one of the example use-cases that will be part of the theoretical text as well, -[the 'pinwall' demo][pinwall] \[[`5ec1fe2`][5ec1fe2]\]. - -The Pinwall example renders all its children as resizeable and movable boxes that can be freely positioned on a canvas: - -<mmm-embed nolink path="demo"></mmm-embed> - -Any changes to the box positions and sizes are saved persistently as a `pinwall_info` facet on each child. -For example the size and coordinates of the image can be found at [`image/pinwall_info: text/json`][info]. - -Rendering the children themselves is pretty easy: - -```moon -import article, div from require 'mmm.dom' -import convert from require 'mmm.mmmfs.conversion' - -update_info = (fileder, x, y, w, h) -> - info = (fileder\get 'pinwall_info: table') or x: 100, y: 100, w: 300, h: 300 - info.x = x if x - info.y = y if y - info.w = w if w - info.h = h if h - - json = convert 'table', 'text/json', info, fileder, 'pinwall_info' - fileder\set 'pinwall_info: text/json', json - -=> - observe = ... -- (ommited - calls `update_info` when child is resized) - - children = for child in *@children - info = (child\get 'pinwall_info: table') or x: 100, y: 100, w: 300, h: 300 - wrapper = div { - style: - position: 'absolute' - -- (more styling omitted here) - - left: "#{info.x}px" - top: "#{info.y}px" - width: "#{info.w}px" - height: "#{info.h}px" - - -- handle for moving the child - div { - style: - -- (styling omitted here) - - onmousedown: ... -- (omitted) - } - - -- child content - div { - style: - width: '100%' - height: '100%' - background: 'var(--white)' - - (child\gett 'mmm/dom') - } - } - - observe wrapper, child - - wrapper - - children.style = { - width: '1000px' - height: '500px' - } - - children.onmouseup = ... -- (omitted) - children.onmousemove = ... -- (omitted) - children.onmouseleave = ... -- (omitted) - - article children -``` - -[The rest of the code][source] is just about catching the events when the mouse is clicked/release/moved and when a child is resized, -and then calling `update_info` as appropriate. - -[info]: /articles/mmmfs/examples/pinwall/image/pinwall_info:%20text/html+interactive -[pinwall]: /articles/mmmfs/examples/pinwall/ -[source]: https://git.s-ol.nu/mmm/blob/5ec1fe2fc943ad4123fac138de70d4152e8b341d/root/articles/mmmfs/examples/pinwall/text%24moonscript%20-%3E%20fn%20-%3E%20mmm%24dom.moon -[5ec1fe2]: https://git.s-ol.nu/mmm/blob/5ec1fe2fc943ad4123fac138de70d4152e8b341d/ diff --git a/root/articles/mmmfs/ba_log/2019-12-20/text$markdown+sidenotes.md b/root/articles/mmmfs/ba_log/2019-12-20/text$markdown+sidenotes.md deleted file mode 100644 index 3de1a9c..0000000 --- a/root/articles/mmmfs/ba_log/2019-12-20/text$markdown+sidenotes.md +++ /dev/null @@ -1,39 +0,0 @@ -In the last three days I have been working extensively on support for sidenotes and academic referencing, -inspired by Edward Tufte's style of publishing (as seen in *Beatiful Evidence* and documented in [tufte-css][tufte-css]. - -To this end margin-notes have been implemented in the CSS styling of the page using two classes, `sidenote` and -`sidenote-container`, which are to be applied to individual sidenotes and the containing document respectively. -Sidenotes are then pulled out of their surrounding context using `position: absolute` and placed in a margin that is -left free by `sidenote-container`. - -Inside of markdown files, sidenotes can then be added simply using basic HTML, like so: - -```md -<div class="sidenote">additional information to be found on the margin</div> -An example paragraph of text, describing something. -``` - -Which will render like this: - -> <div class="sidenote">additional information to be found on the margin</div> -> An example paragraph of text, describing something. - -Additionally, conversions from `text/bibtex`, a reference specification format, to `mmm/dom` have been added, that -create citations using the metadata available in the BibTeX file. - -For example the following BibTeX is rendered like this: - -<mmm-embed nolink path="../../references/inkandswitch" facet="markdown"></mmm-embed> - -> <mmm-embed wrap="raw" path="../../references/inkandswitch"></mmm-embed> - -I also added a special override that links to -BibTeX files by placing the citation in a sidenote, and adding a footnote indicator in-text. - -There is also a handy convert that turns ACM Digital Library links into URLs that directly return the BibTeX file, -which allows me to cite the links directly without manually adding the BibTeX information to my document. - -All of this is implemented in the `cites` plug-in: [`cites.moon`][cites.moon]. - -[cites.moon]: https://git.s-ol.nu/mmm/blob/ba/mmm/mmmfs/plugins/cites.moon -[tufte-css]: https://edwardtufte.github.io/tufte-css/ diff --git a/root/articles/mmmfs/ba_log/intro: text$markdown.md b/root/articles/mmmfs/ba_log/intro: text$markdown.md deleted file mode 100644 index 37366cf..0000000 --- a/root/articles/mmmfs/ba_log/intro: text$markdown.md +++ /dev/null @@ -1,4 +0,0 @@ -The following pages document the development of the `mmmfs` system described above in the form of a project log. -Please note that the log has been written primarily for viewing using a web browser, and as such makes extensive use of -hyperlinking, and also includes some videos that cannot be reproduced in print. It is therefore recommended to view the -live version of the log online at the following address: [s-ol.nu/ba/log](https://s-ol.nu/ba/log). diff --git a/root/articles/mmmfs/ba_log/print: text$moonscript -> fn -> mmm$dom.moon b/root/articles/mmmfs/ba_log/print: text$moonscript -> fn -> mmm$dom.moon deleted file mode 100644 index 97ebd4c..0000000 --- a/root/articles/mmmfs/ba_log/print: text$moonscript -> fn -> mmm$dom.moon +++ /dev/null @@ -1,19 +0,0 @@ -import div, section, h1, h2, hr from require 'mmm.dom' -import link_to from (require 'mmm.mmmfs.util') require 'mmm.dom' -import ropairs from require 'mmm.ordered' - -=> - div { - class: 'print-ownpage' - - h1 (link_to @, "appendix: project log"), id: 'ba-log' - @gett 'intro: mmm/dom' - table.unpack for post in *@children - continue if post\get 'hidden: bool' - - section { - hr! - h2 link_to post, post\gett 'name: mmm/dom' - (post\gett 'mmm/dom') - } - } diff --git a/root/articles/mmmfs/ba_log/start/text$markdown.md b/root/articles/mmmfs/ba_log/start/text$markdown.md deleted file mode 100644 index d56130b..0000000 --- a/root/articles/mmmfs/ba_log/start/text$markdown.md +++ /dev/null @@ -1,25 +0,0 @@ -The system described in the thesis and subject to the following blog posts was partially pre-existing to the work -done as part of the thesis and thesis project. The `mmmfs` system was originally developed as the software for my -personal website, with development beginning around May 2018. In this phase of development the website and system -changed shape drastically multiple times. In the following paragraphs I will describe the state of the project at -the last revision prior to the beginning of the thesis project, as it existed before the 7th of October 2019. -All developments since this revision have been tracked in the following project log entries, and are to be considered -the practical contributions to the thesis project. - -In this revision, the system existed as a tool to produce a static website representing the contents of the `mmmfs` -system in browsable HTML format. The tool had to be run manually after changes, or using a build tool like [`tup`][tup]. -A primitive *Inspector* tab was present in the HTML output and allowed viewing the raw `mmmfs` contents, but all editing -of content had to be done with external tools and in the external file-system. To see changes in the browser, a -compilation phase had to be triggered and completed, and consecutively the page reloaded in the browser. There was no -server-side component that could convert content or store changes for clients. The tool could only accept content from -the filesystem, not from zip archives or SQLite databases. The conversion algorithm used was more naive and was not -able to track cost values, which meant that some more advanced conversions couldn't be implemented. - -There was already an example implementation of a simple slideshow present, but due to the lack of editing capabilities -there was no pinwall examle. There was also no support for side- or marginnotes, or academic referencing and citations. -There also was no support for [`mermaid`][mermaid] diagrams or JSON-encoded data, and there was no plugin interface. -The HTML template, CSS styling and JS runtime were all built and tracked outside of the `mmmfs` system itself. -There was no unit tests for any of the project. - -[tup]: http://gittup.org/tup/ -[mermaid]: https://mermaidjs.github.io/ diff --git a/root/articles/mmmfs/ba_log/text$moonscript -> fn -> mmm$dom.moon b/root/articles/mmmfs/ba_log/text$moonscript -> fn -> mmm$dom.moon deleted file mode 100644 index b6b0756..0000000 --- a/root/articles/mmmfs/ba_log/text$moonscript -> fn -> mmm$dom.moon +++ /dev/null @@ -1,15 +0,0 @@ -import div, h1, ul, li from require 'mmm.dom' -import link_to from (require 'mmm.mmmfs.util') require 'mmm.dom' -import ropairs from require 'mmm.ordered' - -=> - div { - h1 (link_to @, "appendix: project log"), id: 'ba-log' - @gett 'intro: mmm/dom' - ul do - posts = for post in *@children - continue if post\get 'hidden: bool' - li link_to post, post\gett 'name: mmm/dom' - - posts - } |
