Problem with credentials prompt

[ewohnlich]

If I enter packages with an HTTPS source that requires authentication, I will be prompted to enter a username and password as expected. However mxdev continues to output text on updating the other packages, instead of waiting for an answer to the prompt. This is particularly problematic if I have multiple packages that require authentication as their prompts seem to overlap. It will, for example, want me to enter the username for package1 then the username for package2, then the passwords. Very difficult to tell whether I'm being prompted for username or a password, and for which package.

Workaround is to of course switch to SSH clones, but seemed worth mentioning.

[jensens]

try:

[settings]
threads=1

[jensens]

Root Cause Analysis

The issue occurs because mxdev processes VCS operations in parallel using multiple threads (default: 4 threads). When a git subprocess needs user input (e.g., HTTPS credentials), it prompts directly to the TTY. Meanwhile, other threads continue their VCS operations and output to the same TTY, causing:

1. Prompts get mixed with other output and scroll away
2. Multiple credential prompts overlap (username for package1, username for package2, then passwords)
3. User can't tell what they're being prompted for

Current workaround: threads=1 (defeats the purpose of parallel operations)

Why we can't use traditional locking: The git subprocess controls its own stdin/stdout. Python code can't easily detect when git is prompting or intercept its TTY access.

---

Proposed Solution: Two-Phase Checkout

Process packages in two phases to intelligently handle interactive operations:

Phase 1: Serial execution of HTTPS packages

• Detect packages with HTTPS URLs (likely to need credentials)
• Process these serially (threads=1) at the start
• User sees clean, uninterrupted prompts
• Git credential helper caches credentials

Phase 2: Parallel execution of remaining packages

• Process SSH/local packages in parallel (unlikely to prompt)
• Use full thread pool for speed
• Cached credentials from Phase 1 mean no more prompts

Benefits

✅ No new dependencies - uses only stdlib
✅ Backward compatible - no config changes needed
✅ Automatic detection - works transparently
✅ Works with git credential helpers - credentials cached after first prompt
✅ Preserves parallelism - SSH and local repos still process in parallel
✅ Clean UX - credential prompts are clear and sequential

Implementation Details

def checkout(self, packages, **kwargs):
    # Separate HTTPS (might prompt) from others (won't prompt)
    https_packages = []
    other_packages = []
    
    for name in packages:
        source = self.sources[name]
        url = source.get("url", "")
        if url.startswith("https://"):
            https_packages.append(name)
        else:
            other_packages.append(name)
    
    # Phase 1: Process HTTPS packages serially (save original threads setting)
    if https_packages:
        logger.info("Processing %d HTTPS packages serially...", len(https_packages))
        original_threads = self.threads
        self.threads = 1  # Serial execution
        self._checkout_packages(https_packages, **kwargs)
        self.threads = original_threads
    
    # Phase 2: Process remaining packages in parallel
    if other_packages:
        logger.info("Processing %d packages in parallel...", len(other_packages))
        self._checkout_packages(other_packages, **kwargs)

Alternative Considered: URL Detection in process()

Instead of splitting in checkout(), could detect HTTPS URLs when queueing and force serial processing dynamically. However, the two-phase approach is cleaner and more predictable.

---

Questions for Review

1. Should we also handle HTTPS URLs in the update() method similarly?
2. Should this behavior be configurable (e.g., smart-threading=true)?
3. Are there other URL patterns that might prompt (e.g., specific git+https formats)?

I'm ready to implement this if the approach looks good!

[jensens]

Updated Implementation Plan

Based on feedback, the solution will:

1. ✅ Apply two-phase processing to both checkout() and update() methods (HTTPS can prompt during updates too)
2. ✅ Make it configurable with smart-threading setting (default: true)

Configuration

[settings]
# Enable intelligent threading that processes HTTPS packages serially
# to avoid overlapping credential prompts (default: true)
smart-threading = true

When smart-threading = true:

• HTTPS packages are processed serially first (clean prompts)
• Then SSH/local packages are processed in parallel (full speed)

When smart-threading = false:

• Original behavior (all packages processed in parallel)
• Useful if you have credential helpers configured and never see prompts

Implementation

Both WorkingCopies.checkout() and WorkingCopies.update() will use the same two-phase logic when smart-threading is enabled.

---

Proceeding with implementation now...