About V4, API, architecture and package management

[Dexmachi]

In my last discussion, I talked about how the Pyinfra architecture is quite coupled with UI elements. I'd like to start a discussion about how the project's architecture could evolve to make its API surface more stable for the future

Well, first things first: My use-case

I currently utilize Pyinfra as the core execution API for my own system management project (akin to a platform, with secret management and such). That being said, when I utilized Pyinfra, I realized that it has a tight coupling with UI elements:

   class PyinfraFactLogHandler(logging.Handler):
       """Gets command logs from pyinfra's logger and logs them to the database."""

       def emit(self, record):
           msg = record.getMessage()
           op_hash = op_hash_context.get()
           if not ***Telemetry._limani_plugin:
               raise RuntimeError("Limani plugin is not loaded.")

           # --- Diff handling (remains in-memory for the duration of exactly one op) ---

           if record.levelname == "INFO" and op_hash:
               is_diff_start = "Will modify" in msg
               is_part_of_active_diff = op_hash in ***Telemetry._active_diffs

               if is_diff_start:
                   ***Telemetry._active_diffs.add(op_hash)
                   is_part_of_active_diff = True

               if is_part_of_active_diff:
                   if op_hash not in ***Telemetry._diff_log_buffer:
                       ***Telemetry._diff_log_buffer[op_hash] = []
                   cleaned_msg = ***Telemetry._strip_ansi_codes(msg)

                   if "]" in cleaned_msg:
                       cleaned_msg = cleaned_msg.split("]", 1)[-1].strip()
                   cleaned_msg = ***Telemetry._sanitize_diff_text(cleaned_msg)
                   ***Telemetry._diff_log_buffer[op_hash].append(cleaned_msg)

                   if "Success" in msg or "No changes" in msg:
                       ***Telemetry._active_diffs.discard(op_hash)
                   return

           is_fact_gathering = "Getting fact:" in msg and record.levelname == "DEBUG"
           is_command_running = (
               "--> Running command" in msg and record.levelname == "DEBUG"
           )

           run_id = ***Telemetry._run_id
           if not run_id or not (is_fact_gathering or is_command_running):
               return

           context, command = None, None
           if is_fact_gathering:
               context = "fact_gathering"
               command = msg.split(":", 1)[-1].strip()

           elif is_command_running:
               try:
                   context_part, command_part = msg.split(":", 1)
                   context_text = context_part.split("-->")[1].strip()
                   context = context_text.replace(" ", "_").lower()
                   command = command_part.strip()
               except (ValueError, IndexError):
                   context = "running_command"
                   command = msg

           if context and command:
               ***Telemetry._stream_***_event(
                   {
                       "type": "fact",
                       "timestamp": record.created,
                       "log_level": record.levelname,
                       "context": context,
                       "command": command,
                   }
               )

               if ***Telemetry._db_queue:
                   log_data = {
                       "run_id": run_id,
                       "timestamp": record.created,
                       "log_level": record.levelname,
                       "context": context,
                       "command": command,
                   }
                   ***Telemetry._db_queue.put(
                       (***Telemetry._limani_plugin.insert_fact_log, [], log_data)
                   )

As per the code above, I have to capture Pyinfra's logging output in order to capture diffs, commands, and facts. These could be put in the OperationMeta object as well as the logger, in order to make API usage easier

This is a coupling that makes API usage more brittle and susceptible to unsuspecting breaking changes (e.g., different strings being passed to the logger)

I know this is already a known issue, and it started being worked on by @wowi42 in #1616, making an amazing step forward by removing all calls to click in the API

looking forward to V4

I've realized that @JakkuSakura is doing huge work on #1482 by dropping paramiko and using asyncio instead. This can be awesome for quite a few different reasons, but it still is a major breaking change that should be looked at closely.

I think we could use this breaking change window to discuss API decoupling and how to manage packaging for Pyinfra. Currently, it installs both the CLI and the API all in one, making packaging it with different interfaces harder (requiring additional dependencies such as click even when running purely headless).

As such, I'd like to make a few questions for the maintainers, in order to be as helpful as I can be with this project, and I believe can help other contributors to make the project grow:

1. What is the current architectural philosophy regarding Pyinfra as a programmatic API vs. a CLI tool?
2. What are the current doubts or blockers with V4's move to asyncio, so we can help find workarounds?
3. What is the current project's roadmap?