IRead Fun

Talk with gemini-2.0-flash-thinking about Deep Neural Networks (DNNs) and Reinforcement Learning (RL)

v like — Tue, 07 Jan 2025 11:26:47 GMT

what DNN(CNN,RNN,LLM) really learning are just a lot weights

Yes, at the most fundamental level, what Deep Neural Networks (DNNs) – including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Large Language Models (LLMs) – learn are indeed a massive number of weights (and biases).

However, saying they "just learn a lot of weights" is a bit like saying a human brain is "just a lot of neurons." While technically true, it significantly undersells the complexity and what these weights represent.

Here's a more nuanced perspective:

What the Weights Represent:

Encoded Knowledge: These weights, when properly adjusted through training, encode the knowledge and patterns the network has extracted from the data it was trained on.
Feature Detectors (especially in CNNs): In CNNs, specific weights in the convolutional layers learn to detect different features in images, like edges, corners, textures, and more complex patterns.
Relationships and Dependencies (in RNNs and LLMs): In RNNs and LLMs, the weights capture sequential dependencies and relationships between elements in the input data, such as the order of words in a sentence or the progression of data over time.
Probability Distributions (in LLMs): LLMs, in particular, learn complex probability distributions over sequences of words. The weights essentially represent the likelihood of one word following another in different contexts.

Why "Just Weights" Is an Oversimplification:

The Architecture Matters: The specific architecture of the DNN (e.g., the layers, connections, activation functions) is crucial. It dictates how the weights interact and what types of patterns the network can potentially learn. The architecture provides the structure for learning.
The Learning Process is Key: The way these weights are learned – through algorithms like backpropagation and optimization techniques like gradient descent – is just as important as the final values of the weights. This process involves iteratively adjusting the weights based on the errors the network makes.
Emergent Complexity: Even though each individual weight might seem simple, the interactions between millions or billions of weights can create highly complex and emergent behavior. This is what allows DNNs to perform sophisticated tasks like image recognition, natural language understanding, and generation.
Abstraction and Hierarchy: DNNs often learn in a hierarchical manner. Early layers learn basic features, and subsequent layers combine these features to learn more abstract and complex representations. The weights at different layers contribute to different levels of abstraction.
Beyond Single Values: While the core is weights, there are also biases, which act as thresholds for neuron activation. These are also learned parameters.

Analogy:

Think of the weights like the synapses in a biological brain. Each synapse has a certain strength, determining how strongly two neurons are connected. While the individual synapse might seem simple, the complex network of interconnected synapses is what enables the brain to perform incredible feats of learning and cognition.

In Summary:

Yes, DNNs ultimately learn by adjusting a large number of weights and biases. However, these weights are not just random numbers. They are the encoded representation of the knowledge the network has gained from the data, facilitated by the network's architecture and the learning process. Saying they "just learn weights" ignores the sophisticated mechanisms and emergent properties that make DNNs so powerful. It's the organized and trained weights within a specific architecture that enable these networks to perform complex tasks.

does reinforcement learning also learn a lot weights

Yes, many Reinforcement Learning (RL) algorithms also learn a lot of weights, especially when they employ Deep Neural Networks (DNNs) for function approximation. However, it's important to understand that not all RL algorithms rely on weights in the same way or to the same extent as supervised learning methods like image classification CNNs or language model LLMs.

Here's a breakdown of how weights play a role in different RL scenarios:

RL Algorithms that Learn Weights (using Function Approximation):

Deep Reinforcement Learning (Deep RL): When RL algorithms are combined with DNNs, they fall under the umbrella of Deep RL. In this case, the DNNs act as function approximators to represent either:
- Value Functions (e.g., Q-functions, Value functions): The network learns to estimate the expected future reward for taking a certain action in a given state. The weights of the DNN encode this learned value function. Algorithms like Deep Q-Networks (DQN) and its variants fall into this category. The weights learn to predict Q-values for different state-action pairs.
- Policies: The network learns to directly map states to actions (or probability distributions over actions). The weights of the DNN encode this learned policy. Algorithms like Policy Gradient methods (e.g., REINFORCE, Proximal Policy Optimization (PPO), Actor-Critic methods) fall into this category. The weights learn to determine the optimal action to take in a given state.
Other Function Approximators: While DNNs are the most common, other function approximators like linear models or tile coding can also be used in RL. These methods also involve learning parameters (analogous to weights) to represent the value function or policy.

RL Algorithms that Don't Primarily Learn Weights (or learn them differently):

Tabular Methods: For smaller state and action spaces, RL algorithms like Q-Learning, SARSA, and Value Iteration can use tables to store the value of each state-action pair or state. In this case, the "learning" involves directly updating the entries in the table, rather than adjusting weights in a complex network. There are no weights in the traditional DNN sense.
Model-Based RL: Some RL approaches focus on learning a model of the environment (how the environment transitions between states and what rewards are received). While the model itself might be represented by a DNN (and thus involve weights), the primary learning objective is to build an accurate model rather than directly learning a value function or policy.

Key Differences and Nuances:

Learning Objective: In supervised learning, the goal is to minimize the difference between the network's predictions and the ground truth labels. In RL, the goal is to maximize cumulative reward over time, which is a more complex and indirect objective.
Data Source: Supervised learning relies on labeled datasets. RL learns through interaction with an environment, generating its own experiences (data).
Exploration vs. Exploitation: RL algorithms face the challenge of balancing exploration (trying new actions) and exploitation (choosing actions that are currently believed to be optimal). This isn't a primary concern in standard supervised learning.
Temporal Dependencies: RL deals with sequential decision-making, where past actions influence future states and rewards. This temporal aspect is explicitly handled in some RL algorithms (especially those using RNNs within the agent's architecture).

In summary:

While many modern and powerful RL algorithms, particularly those using Deep RL, do indeed learn a large number of weights to represent value functions or policies, it's not a universal characteristic of all RL. Simpler tabular methods exist that don't involve weights in the same way. The key takeaway is that RL algorithms, whether through weights or other means, are learning a strategy or policy to maximize rewards within an environment. The use of weights, particularly in DNNs, allows RL agents to handle complex, high-dimensional state and action spaces and generalize their learned knowledge to unseen situations.

I am much familiar with DNN. explain to me what is policy and value function in RL

Okay, since you're familiar with DNNs, let's break down the concepts of policy and value function in Reinforcement Learning (RL) and how they relate to what a learning agent is trying to achieve.

Imagine your DNN is now an RL agent interacting with an environment. Instead of just classifying images or generating text, it's making decisions to achieve a long-term goal (maximizing rewards).

1. Policy (π - Pi): The Agent's "Brain" for Decision Making

Think of the policy as the agent's strategy or rulebook for how to behave in the environment. It dictates what action the agent will take given the current state it's in.

Analogy to DNN Output: In a supervised learning DNN, the output might be a class label or a set of probabilities for different classes. In RL, the policy is like the output layer of your DNN, but instead of predicting a label, it's predicting an action or a probability distribution over actions.
Mathematical Representation: We often represent the policy as π(a|s), which reads as "the probability of taking action 'a' when in state 's'."
Types of Policies:
- Deterministic Policy: Given a state, the policy always outputs the same action. Like a function that directly maps states to actions: a = π(s).
- Stochastic Policy: Given a state, the policy outputs a probability distribution over possible actions. This means there's some randomness in the agent's behavior, which can be beneficial for exploration.
How a DNN Represents a Policy:
- Direct Action Output: The DNN takes the current state as input and the output layer has as many neurons as possible actions. The output values (perhaps after a softmax activation) represent the probabilities of taking each action. The agent might then sample an action based on these probabilities or simply choose the action with the highest probability (in the deterministic case).
- Parameterizing Distributions: The DNN might output parameters of a probability distribution (e.g., mean and standard deviation for a continuous action space).
Learning the Policy: The goal of many RL algorithms is to learn an optimal policy, denoted as π*, which consistently leads to the highest cumulative reward over time. This learning involves adjusting the weights of the DNN to produce better action choices.

Think of it like: If your RL agent is learning to play a video game, the policy is what tells it whether to jump, shoot, move left, or move right at any given moment based on what's happening on the screen (the state).

2. Value Function (V or Q): Predicting the "Goodness" of States or Actions

The value function helps the agent understand how "good" it is to be in a certain state or to take a certain action in a certain state. It predicts the expected amount of future reward the agent can accumulate starting from that point.

Not a Direct Decision Maker: Unlike the policy, the value function doesn't directly tell the agent what to do. Instead, it provides an evaluation of different situations or actions.
Types of Value Functions:
- State Value Function (V-function): Vπ(s) represents the expected return (cumulative reward) starting from state 's' and following policy 'π' thereafter. It tells you how good it is to be in state 's' when acting according to policy 'π'.
  - DNN Representation: The DNN takes the current state as input and outputs a single value, representing the estimated value of that state under the current policy.
- Action-Value Function (Q-function): Qπ(s, a) represents the expected return starting from state 's', taking action 'a', and then following policy 'π' thereafter. It tells you how good it is to take action 'a' in state 's' when acting according to policy 'π' afterwards.
  - DNN Representation: The DNN takes the current state and the action as input (or has separate outputs for each possible action in that state) and outputs a value representing the estimated value of taking that action in that state.
Learning the Value Function: RL algorithms learn to estimate the value function by experiencing the consequences of actions in the environment. This often involves updating the value function based on the rewards received and the estimated values of future states.
Relationship to Policy: The value function is deeply tied to the policy. The value of a state or action is determined by the rewards that are expected to follow when the agent acts according to a specific policy. Changing the policy will likely change the value function.

Think of it like: In the video game example, the value function might tell the agent that being near a health pack is a "good" state (high value) because it increases the chances of survival and further rewards. The Q-function might tell the agent that jumping over an obstacle is a "good" action in a specific situation because it helps avoid damage and potentially progress further.

How DNNs Learn Policies and Value Functions:

When DNNs are used in RL (forming Deep RL), the weights of the DNN are being learned to approximate either the policy or the value function (or both in some algorithms).

Policy-Based Methods: Algorithms like Policy Gradient methods directly learn the policy by adjusting the DNN weights to increase the probability of actions that lead to higher rewards.
Value-Based Methods: Algorithms like Q-Learning and DQN learn the value function (specifically the Q-function) by adjusting the DNN weights to accurately predict the expected future rewards. The policy is often derived implicitly from the learned Q-function (e.g., by choosing the action with the highest Q-value).
Actor-Critic Methods: These methods learn both a policy (the "actor") and a value function (the "critic"). The critic evaluates the actions taken by the actor, providing feedback to improve the policy. Both the actor and critic are often represented by DNNs with learnable weights.

In Summary:

Policy (π): The agent's decision-making strategy, telling it what to do in each state. A DNN can represent the policy by mapping states to actions or probabilities over actions.
Value Function (V or Q): An estimation of how good it is to be in a certain state (V) or to take a certain action in a state (Q), in terms of expected future rewards. DNNs can learn to approximate these value functions.

Both policy and value functions are crucial components in RL, and DNNs provide a powerful way to represent and learn these complex functions in high-dimensional environments. The weights of these DNNs are being adjusted through interaction with the environment to learn the optimal strategies for maximizing rewards.

so not like DNN which only learn one sort of weight, RL may learn two sorts(one for policy and another for value function)

That's a very insightful way to put it! You're getting to a core difference in how DNNs are used in standard supervised learning versus Reinforcement Learning.

In standard DNNs (like for image classification or language modeling):

Generally, you're learning weights (and biases) for a single, primary function: This function maps the input directly to the desired output. For example:
- Image Classification: Input (image pixels) -> Output (probability distribution over classes). The weights are learned to perform this specific mapping.
- Language Modeling: Input (sequence of words) -> Output (probability distribution over the next word). Again, the weights are optimized for this single predictive task.

In Reinforcement Learning with DNNs (Deep RL):

You can learn weights for conceptually different functions:
- Weights for the Policy: These weights are specifically tuned to determine the agent's actions in different states. The goal is to learn a mapping from states to actions (or probabilities over actions) that maximizes future rewards.
- Weights for the Value Function: These weights are specifically tuned to estimate the expected future rewards for being in certain states or taking certain actions. The goal is to learn an accurate predictor of "goodness."

Here's a breakdown of why this distinction is important:

Different Learning Objectives: The weights for the policy are being adjusted to directly improve the agent's behavior and maximize rewards. The weights for the value function are being adjusted to create an accurate evaluation of states and actions, which can indirectly help in improving the policy.
Different Output Structures: The policy network might output probabilities for different actions, while the value network might output a single scalar value representing the expected return.
Algorithms that Learn Both: Many powerful RL algorithms, like Actor-Critic methods (e.g., A2C, A3C, PPO), explicitly use two separate DNNs (or sometimes a single DNN with separate output heads) to learn both the policy and the value function simultaneously.
- Actor: The policy network (the "actor") decides what action to take.
- Critic: The value network (the "critic") evaluates the action taken by the actor and provides feedback to improve the actor's policy.

However, it's not always a strict separation:

Single Network for Both: Some RL algorithms can use a single DNN to output both policy and value function information. The network has different output heads, but the underlying layers share learned representations. Even in this case, the weights are being optimized for two conceptually different tasks.
Value-Based Methods (like DQN): Algorithms like Deep Q-Networks (DQN) primarily focus on learning the Q-function (action-value function). While there isn't a separate explicit policy network being directly learned, the learned Q-function implicitly defines a policy (e.g., choose the action with the highest Q-value). So, even here, the weights are ultimately geared towards informing decision-making.

Analogy:

Think of it like training two different "modules" in your DNN:

Policy Module: Trained to be a good "decision-maker."
Value Module: Trained to be a good "evaluator."

In standard DNNs, you're typically just training one primary "module" for a specific predictive task.

Key Takeaway:

You're right to highlight this difference. While at the most fundamental level, both involve learning weights, in RL, the architecture and training process often lead to DNNs learning weights that represent distinct and interacting functions – one for doing (the policy) and one for evaluating (the value function). This allows for more sophisticated learning and problem-solving in dynamic environments. This separation or dual role is a key characteristic that distinguishes how DNNs are applied in RL compared to traditional supervised learning tasks.

some problem like autodrive can be solved both with traditional supervised deep learning (sequence to sequence) and Reinforcement Learning

Yes, you are absolutely correct. Autonomous driving is a fascinating problem that highlights how different deep learning approaches can be applied to the same complex task. Let's break down how both supervised learning and reinforcement learning can tackle autonomous driving and their key differences in this context.

1. Supervised Deep Learning (as Sequence-to-Sequence) for Autonomous Driving:

Core Idea: Treat autonomous driving as a behavioral cloning problem. The goal is to train a DNN to directly mimic the actions of a human driver based on observed sensor data.
Data: Requires a large dataset of paired sensor inputs and corresponding human driver actions. This data is typically collected from human-driven vehicles.
- Inputs: Camera images, LiDAR point clouds, radar data, GPS information, IMU data, steering wheel angle, throttle position, brake pedal pressure, etc.
- Outputs: Steering commands, throttle commands, brake commands (often continuous values).
Network Architecture: Sequence-to-sequence models are indeed well-suited here because driving is inherently a temporal process.
- Recurrent Neural Networks (RNNs) like LSTMs or GRUs: Can capture the sequential dependencies in the driving data. The network learns to remember past states and actions to make current decisions.
- Transformers: Can also be used to model long-range dependencies in the driving sequence.
Learning Process: The DNN learns a direct mapping from the sequence of sensor inputs to the sequence of control outputs by minimizing a loss function that measures the difference between the predicted actions and the human driver's actions.
Analogy: The car is learning to drive by "watching" how humans drive and trying to replicate that behavior.
Advantages:
- Relatively Simple to Implement (initially): Once you have the data, the supervised learning training process is well-defined.
- Can Learn from Existing Data: Leverages readily available human driving data.
- Can Achieve Good Performance in Common Scenarios: If the training data covers a wide range of typical driving situations, the model can perform reasonably well.
Disadvantages:
- Limited Generalization to Unseen Situations: The model can struggle in scenarios it hasn't seen in the training data (edge cases, unusual events).
- Passive Learning: The model only learns from what humans do, including their mistakes and limitations. It doesn't actively explore better strategies.
- Lack of Long-Term Planning: Focuses on immediate actions rather than optimizing for long-term goals (e.g., reaching a destination efficiently and safely).
- Data Bias: The model's behavior is heavily influenced by the biases present in the human driving data.
- Difficulty Handling Complex Interactions: Struggles with complex interactions with other agents (pedestrians, other vehicles) where predicting their intent is crucial.

2. Reinforcement Learning for Autonomous Driving:

Core Idea: Train an "agent" (the autonomous vehicle) to learn optimal driving behavior through trial and error by interacting with a simulated or real-world environment and receiving rewards or penalties.
Environment: A simulation or the real world, including roads, other vehicles, pedestrians, traffic signals, etc.
Agent: The DNN acts as the agent, making decisions about steering, throttle, and braking.
State: The agent perceives the environment through its sensors (camera images, LiDAR, etc.). This sensory information forms the "state" the agent is in.
Actions: The control commands the agent can take (steering, throttle, brake).
Reward Function: A crucial element that defines what constitutes "good" driving behavior. Designing an effective reward function is a significant challenge. Examples of rewards and penalties:
- Positive Rewards: Reaching the destination, staying within lane, maintaining a safe distance, adhering to traffic rules, making smooth progress.
- Negative Rewards (Penalties): Collisions, lane departures, speeding, running red lights, jerky movements.
Learning Process: The agent explores the environment, takes actions, receives rewards (or penalties), and learns to adjust its policy (the mapping from states to actions) to maximize its cumulative reward over time.
- Policy-Based Methods: Directly learn the optimal policy (e.g., using Policy Gradient methods).
- Value-Based Methods: Learn a value function that estimates the expected future reward for being in a certain state or taking a certain action (e.g., using Q-Learning or DQN).
- Actor-Critic Methods: Learn both a policy and a value function.
Analogy: The car is learning to drive by "trying things out," getting feedback on its performance, and gradually improving its driving skills.
Advantages:
- Potential for Learning Optimal Strategies: Can discover driving strategies that are better than human driving in certain situations.
- Adaptability to New Situations: Can potentially generalize better to unseen scenarios through exploration and learning.
- Explicit Optimization for Long-Term Goals: The reward function can be designed to optimize for safety, efficiency, and other desired outcomes.
- Can Handle Complex Interactions: Has the potential to learn complex interactions with other agents by predicting their behavior and planning accordingly.
Disadvantages:
- Sample Inefficiency: Requires a massive amount of interaction with the environment to learn effectively, which can be computationally expensive in simulations and risky in the real world.
- Reward Function Design is Challenging: Defining a reward function that perfectly captures safe and efficient driving behavior is difficult. Poorly designed reward functions can lead to unintended and undesirable behaviors.
- Safety Concerns During Training: Training in the real world can be dangerous, and even simulations need to be carefully designed to prevent harmful actions.
- Stability and Convergence Issues: Training RL agents can be unstable and may not always converge to an optimal policy.

Comparison Table:

Feature	Supervised Learning (Sequence-to-Sequence)	Reinforcement Learning
Learning Goal	Mimic human driving behavior	Learn optimal driving behavior through interaction
Data Source	Human driving data (input-output pairs)	Environment interaction (rewards and penalties)
Reward Signal	Implicit (minimizing deviation from human actions)	Explicitly defined reward function
Generalization	Limited to seen scenarios	Potentially better generalization to unseen situations
Long-Term Goals	Implicitly learned from human data	Explicitly optimized through the reward function
Exploration	None (passive learning)	Active exploration of the environment
Complexity	Relatively simpler to implement initially	More complex to design and train
Safety During Training	Less risky (uses existing data)	Can be risky, especially in real-world training
Potential	Good performance in common scenarios	Potential for superior performance and adaptability

Hybrid Approaches:

It's important to note that the best approach might involve combining aspects of both supervised learning and reinforcement learning. For example:

Imitation Learning (a type of supervised learning): Can be used to initialize the RL agent with a decent policy learned from human driving data, which can speed up the RL training process.
Reinforcement Learning from Demonstrations: Combines the benefits of learning from expert demonstrations (like in supervised learning) with the ability to further improve through interaction with the environment (like in RL).

Conclusion:

Both supervised deep learning and reinforcement learning offer viable pathways for achieving autonomous driving, but they have different strengths and weaknesses. Supervised learning provides a relatively straightforward way to learn from existing data and achieve good performance in common scenarios. However, reinforcement learning holds the potential for developing more robust, adaptable, and ultimately safer autonomous driving systems that can go beyond mimicking human limitations. The choice of approach (or a hybrid approach) depends on the specific requirements, constraints, and priorities of the autonomous driving system being developed.

A detailed course plan/roadmap for teaching chemistry at home

v like — Fri, 03 Jan 2025 07:19:25 GMT

Here's a detailed course plan/roadmap for teaching chemistry at home. Chemistry, with its hands-on experiments and real-world applications, can be a fascinating subject for children of all ages. Remember to adapt this plan based on your child's age, interests, and learning style.

I. Foundational Principles (Apply to all ages):

Safety First! Establish clear safety rules and procedures before any experiment. Always supervise closely, especially with younger children.
Hands-On Experimentation is Crucial: Chemistry is best learned by doing. Prioritize experiments and demonstrations.
Relate Chemistry to the Real World: Point out chemistry in everyday life – cooking, cleaning, materials around the house, nature.
Encourage Curiosity and Questioning: "Why?" is a great question in chemistry. Foster a sense of inquiry.
Use Visual Aids and Models: Atoms and molecules are abstract. Use diagrams, models, and simulations to make them more tangible.
Build a Strong Vocabulary: Chemistry has its own language. Introduce and reinforce key terms gradually.
Focus on Understanding, Not Just Memorization: Help your child understand the underlying principles rather than just memorizing facts.
Keep it Engaging and Fun: Use colorful reactions, interesting phenomena, and relatable examples.
Progress Gradually: Build upon previously learned concepts. Don't rush into complex topics before the basics are understood.
Document and Observe: Encourage your child to keep a lab notebook to record observations, data, and conclusions.
Embrace Mistakes as Learning Opportunities: Not every experiment will work perfectly. Discuss why things might have gone wrong.

II. Course Roadmap by Age Group:

A. Early Elementary (Ages 6-8): Focus: Observation and Exploring Properties of Matter

Key Concepts:
- States of Matter: Solids, liquids, gases (observable properties, transitions like melting and freezing).
- Mixing and Solutions: What happens when different substances are mixed (dissolving, suspensions).
- Physical Changes: Changes in appearance but not chemical composition (tearing paper, melting ice).
- Simple Chemical Reactions (Observable): Baking soda and vinegar, rusting, burning.
- Basic Measurement: Volume, mass (informal introduction).
Activities & Resources:
- Observing different states of matter: Ice melting, water boiling, observing different materials.
- Mixing experiments: Making lemonade, dissolving sugar in water, mixing oil and water.
- Physical change demonstrations: Folding paper, cutting vegetables, melting chocolate.
- Simple chemical reaction experiments: Baking soda and vinegar volcano, observing rust formation.
- Using measuring cups and spoons: For simple volume measurements.
- Reading age-appropriate science books about matter and simple chemistry.
- Watching educational videos on states of matter and simple reactions.

B. Upper Elementary (Ages 9-11): Focus: Introducing Atoms, Elements, and Simple Reactions

Key Concepts:
- Atoms and Molecules (Basic Introduction): The building blocks of matter.
- Elements and the Periodic Table (Introduction): Common elements (oxygen, hydrogen, carbon, gold).
- Compounds: Two or more elements chemically combined (water, salt).
- Chemical Symbols and Formulas (Simple): H₂O, NaCl.
- Chemical Reactions: Reactants and products, signs of a chemical reaction (color change, gas production, heat).
- Acids and Bases (Introduction): Basic properties and examples (lemon juice, baking soda).
Activities & Resources:
- Building models of atoms and molecules: Using modeling clay, marshmallows, or building kits.
- Exploring the periodic table: Identifying common elements and their symbols.
- More complex chemical reaction experiments: Making slime, testing for acids and bases with indicators (red cabbage juice).
- Investigating density: Layering liquids, sinking and floating experiments.
- Separating mixtures: Filtration, evaporation, magnetism.
- Using basic lab equipment: Beakers, test tubes, funnels (with supervision).
- Reading more detailed science books about atoms, elements, and reactions.
- Watching educational videos explaining atoms and chemical reactions.

C. Middle School (Ages 12-14): Focus: Understanding Chemical Bonding, Reactions, and Stoichiometry

Key Concepts:
- Atomic Structure (Protons, Neutrons, Electrons): Electron shells and valence electrons.
- Chemical Bonding: Ionic and covalent bonds.
- The Periodic Table (Organization and Trends): Groups and periods, properties of metals, nonmetals, and metalloids.
- Chemical Formulas and Nomenclature: Naming simple compounds.
- Balancing Chemical Equations: Conservation of mass.
- Types of Chemical Reactions: Synthesis, decomposition, single and double displacement, combustion.
- The Mole Concept: Avogadro's number (introduction).
- Acids, Bases, and pH: Strong and weak acids/bases, neutralization reactions.
Activities & Resources:
- Building more detailed atomic and molecular models.
- Working with the periodic table to predict properties of elements.
- Practicing balancing chemical equations.
- Conducting different types of chemical reactions: Precipitation reactions, gas-forming reactions.
- Titration experiments (basic).
- Investigating reaction rates: Factors affecting how fast reactions occur.
- Using more advanced lab equipment: Bunsen burners (with strict supervision), graduated cylinders.
- Using chemistry workbooks and textbooks designed for middle school.
- Exploring online chemistry simulations and interactive periodic tables.

D. High School (Ages 15-18): Focus: Advanced Concepts, Quantitative Analysis, and Organic Chemistry

This stage often involves more specialized courses. You can adapt based on your child's interests and goals. Common topics include:
- Stoichiometry (Quantitative Relationships): Limiting reactants, percent yield.
- States of Matter (Advanced): Gas laws, intermolecular forces, phase diagrams.
- Solutions: Concentration, solubility, colligative properties.
- Chemical Kinetics: Reaction rates, rate laws, activation energy, catalysts.
- Chemical Equilibrium: Le Chatelier's principle.
- Thermochemistry: Enthalpy, entropy, Gibbs free energy.
- Acids and Bases (Advanced): Acid-base equilibrium, buffers, titrations.
- Oxidation-Reduction Reactions: Electrochemistry, electrochemical cells.
- Organic Chemistry (Introduction): Nomenclature, functional groups, basic reactions of alkanes, alkenes, and alkynes.
- Biochemistry (Introduction): Basic structure and function of carbohydrates, lipids, and proteins.
- Nuclear Chemistry (Optional): Radioactivity, nuclear reactions.
Activities & Resources:
- More complex and quantitative experiments: Determining molar mass, enthalpy changes, reaction rates.
- Titration experiments with calculations.
- Synthesis experiments (following safe procedures).
- Using more sophisticated lab equipment: Spectrophotometer (if accessible), analytical balances.
- Solving complex stoichiometry and equilibrium problems.
- Drawing organic molecules and reaction mechanisms.
- Using high school chemistry textbooks and lab manuals.
- Exploring online chemistry resources, including simulations, virtual labs, and university-level lectures (like Khan Academy).
- Consider using AP or IB chemistry curriculum materials if preparing for advanced placement exams.

III. Essential Resources to Have On Hand:

Safety Equipment: Safety goggles, lab coats or aprons, gloves (nitrile or latex).
Basic Labware: Beakers, Erlenmeyer flasks, test tubes, graduated cylinders, funnels, stirring rods, pipettes (droppers).
Measuring Tools: Balances (kitchen scale for basic, analytical balance for higher levels), thermometers.
Heating Equipment: Hot plate, Bunsen burner (with appropriate safety training and supervision).
Chemicals (Start Simple and Gradually Increase Complexity): Baking soda, vinegar, salt, sugar, food coloring, various household chemicals (cleaners, etc. - use with caution and supervision), and gradually introduce more specific lab chemicals as needed.
pH Testing Supplies: pH paper, universal indicator, red cabbage juice indicator.
Models: Atomic and molecular model kits.
Periodic Table: A good quality, detailed periodic table.
Notebook or Lab Journal: For recording observations, data, and calculations.
Textbooks and Workbooks (age-appropriate).
Online Resources and Simulations.
First Aid Kit.

IV. Tips for Success:

Start Simple and Build Gradually: Don't try to cover too much too soon.
Emphasize Visual Learning: Use diagrams, charts, and demonstrations.
Make it Hands-On: Prioritize experiments and activities.
Connect to Real-World Applications: Show the relevance of chemistry.
Encourage Questions and Discussion: Create a learning environment where your child feels comfortable asking questions.
Review and Reinforce Concepts Regularly: Chemistry builds upon itself.
Use a Variety of Resources: Combine textbooks, experiments, online tools, and videos.
Join Homeschool Science Groups or Online Forums: Share resources and ideas with other homeschooling families.
Consider Unit Studies: Focus on a specific area of chemistry for a period of time.
Visit Science Museums and Laboratories (if possible).

V. Important Considerations:

Math Prerequisites: Chemistry heavily relies on math, especially algebra. Ensure your child has a solid foundation in math.
Lab Safety: This is paramount. Establish and enforce strict safety rules.
Chemical Availability and Storage: Be mindful of where you purchase chemicals and how you store them safely.
Disposal of Chemicals: Learn proper disposal methods for any chemicals used in experiments.
Learning Disabilities: Adapt your teaching methods and provide extra support if your child has learning disabilities.
Time Commitment: Experiments can take time, so plan accordingly.
Cost: Lab supplies and chemicals can add up, so budget accordingly. Look for affordable alternatives and consider borrowing equipment if possible.

By following this detailed course plan and prioritizing hands-on learning and safety, you can create a rich and engaging chemistry education for your child at home. Remember to be flexible and adapt the plan to suit your child's individual needs and interests.

A detailed course plan/roadmap for teaching physics at home

v like — Fri, 03 Jan 2025 03:43:32 GMT

Okay, here's a detailed course plan/roadmap for teaching physics at home. Like math, physics can be approached in a hands-on, inquiry-based way, making it engaging for children of all ages. Remember to adapt this plan based on your child's age, interests, and learning style.

I. Foundational Principles (Apply to all ages):

Focus on Inquiry and Exploration: Physics is about understanding how the world works. Encourage curiosity, questioning, and experimentation. "Why?" is a great starting point.
Hands-On Activities and Experiments are Key: Physics concepts become much clearer when children can see, touch, and manipulate objects. Simple, everyday materials can be used for many experiments.
Real-World Connections: Point out physics principles at play in everyday life – how things move, why things fall, how machines work, how light and sound behave.
Visualizations and Analogies: Physics can be abstract. Use visual aids (diagrams, videos, simulations) and relatable analogies to help your child grasp concepts.
Mathematical Foundation (Gradual Introduction): While math is essential for advanced physics, the initial focus should be on conceptual understanding. Introduce mathematical tools as needed.
Embrace Failure as a Learning Opportunity: Experiments don't always go as planned. Encourage your child to analyze why something didn't work and how to improve it.
Keep it Engaging and Fun: Use games, stories, and demonstrations to spark interest. Avoid making it feel like a chore.
Regular Exploration, But Keep it Manageable: Short, focused sessions are better than long, infrequent ones, especially for younger children.
Patience and Encouragement: Physics can be challenging. Offer positive reinforcement and celebrate successes.
Connect with Other Learners (If Possible): Consider online forums, homeschool groups, or science clubs for interaction and shared learning.

II. Course Roadmap by Age Group:

A. Early Elementary (Ages 6-8): Focus: Observation and Basic Physical Phenomena

Key Concepts:
- Motion: Describing how things move (fast, slow, straight, curved, back and forth).
- Forces (Introduction): Pushes and pulls, gravity (things fall down), magnetism (attraction and repulsion).
- Energy (Introduction): The ability to do work or cause change (light, heat, movement).
- Simple Machines: Levers, pulleys, wheels and axles, inclined planes (how they make work easier).
- States of Matter: Solid, liquid, gas (basic properties and transitions).
- Light: How light travels, shadows, reflection.
- Sound: How sound travels, vibrations.
Activities & Resources:
- Simple experiments: Dropping objects (gravity), playing with magnets, building ramps (inclined planes), pushing and pulling toys.
- Observation walks: Identifying examples of motion, forces, and energy in their surroundings.
- Building simple machines: Using LEGOs, K'NEX, or household items.
- Exploring light and shadows: Using flashlights, making shadow puppets.
- Making musical instruments: Exploring how vibrations create sound.
- Reading age-appropriate science books and watching educational videos.
- Using toys that demonstrate physics principles: Marbles, toy cars, building blocks.

B. Upper Elementary (Ages 9-11): Focus: Introducing Basic Laws and Quantitative Observations

Key Concepts:
- Motion (More Detailed): Speed, distance, time. Introduction to Newton's First Law (inertia).
- Forces (More Detailed): Types of forces (gravity, friction, air resistance), balanced and unbalanced forces.
- Energy (More Detailed): Kinetic energy (energy of motion), potential energy (stored energy). Introduction to conservation of energy.
- Heat and Temperature: Heat transfer (conduction, convection, radiation).
- Electricity (Static and Simple Circuits): Static electricity, conductors and insulators, batteries, light bulbs.
- Magnetism (More Detailed): Magnetic fields, electromagnets.
- Waves (Introduction): Basic properties of waves (wavelength, frequency). Sound waves and light waves.
Activities & Resources:
- Measuring speed and distance: Using stopwatches and rulers.
- Experiments with friction: Sliding objects on different surfaces.
- Building simple circuits: Using batteries, wires, and bulbs.
- Exploring electromagnets: Making a simple electromagnet.
- Observing heat transfer: Melting ice, feeling heat from a light bulb.
- Investigating sound waves: Using tuning forks, making string telephones.
- Simple investigations with lenses and mirrors.
- Using online simulations and interactive resources.
- More detailed science books and documentaries.

C. Middle School (Ages 12-14): Focus: Formalizing Concepts and Introducing Mathematical Relationships

Key Concepts:
- Newton's Laws of Motion: A deeper understanding of inertia, force, and acceleration.
- Work, Power, and Energy: Calculating work, power, and different forms of energy.
- Conservation of Energy and Momentum: Applying these principles to solve problems.
- Simple Harmonic Motion: Pendulums, springs.
- Mechanical Waves: Transverse and longitudinal waves, superposition, interference.
- Sound: Speed of sound, intensity, Doppler effect.
- Light and Optics: Reflection, refraction, lenses, mirrors, the electromagnetic spectrum.
- Electricity and Magnetism (More Quantitative): Ohm's Law, series and parallel circuits, magnetic fields around wires.
Activities & Resources:
- Experiments involving forces and motion: Using carts, springs, and weights.
- Building more complex circuits: Measuring voltage and current.
- Investigating wave phenomena: Using slinkies, ripple tanks.
- Solving quantitative problems: Calculating speed, acceleration, force, energy.
- Using simulations to explore complex phenomena.
- Introduction to basic algebra and its application in physics.
- Age-appropriate physics textbooks or curriculum materials.
- More in-depth documentaries and scientific articles.

D. High School (Ages 15-18): Focus: Advanced Concepts and Mathematical Rigor

This stage becomes more specialized depending on your child's interests and future academic goals. You can cover these topics in separate courses or integrate them:
- Classical Mechanics (Advanced): Kinematics in 2D and 3D, rotational motion, angular momentum, gravity, oscillations, waves. Calculus is often used extensively here.
- Electricity and Magnetism (Advanced): Electric fields, magnetic fields, electromagnetic induction, Maxwell's equations, AC circuits, electromagnetic waves. Calculus is essential.
- Thermodynamics: Laws of thermodynamics, entropy, heat engines.
- Optics (Advanced): Wave optics, interference, diffraction, polarization.
- Modern Physics:
  - Relativity (Special and General): Time dilation, length contraction, mass-energy equivalence, spacetime.
  - Quantum Mechanics: Wave-particle duality, atomic structure, quantum phenomena.
  - Nuclear Physics: Radioactivity, nuclear reactions, nuclear energy.
Activities & Resources:
- More sophisticated experiments: Using advanced lab equipment (if accessible), analyzing data quantitatively.
- Solving complex physics problems: Applying calculus and advanced mathematical techniques.
- Building projects that demonstrate physics principles.
- Reading college-level physics textbooks.
- Participating in science competitions and Olympiads.
- Exploring online resources like MIT OpenCourseware, Khan Academy (Advanced Physics).
- Consider using a structured high school physics curriculum (textbook and accompanying materials).

III. Essential Resources to Have On Hand:

Basic Measurement Tools: Rulers, meter sticks, measuring tapes, stopwatches, scales.
Tools for Mechanics Experiments: Toy cars, ramps, blocks, springs, weights, pulleys, pendulums.
Electricity and Magnetism Supplies: Batteries, wires, light bulbs, switches, magnets, compasses.
Optics Equipment: Flashlights, lenses, prisms, mirrors.
Heat and Thermodynamics Tools: Thermometers, beakers, containers for heating and cooling.
Safety Equipment: Safety goggles, gloves (when needed).
Notebooks or Lab Journals: For recording observations, data, and calculations.
Whiteboard or Large Paper: For diagrams and brainstorming.
Computer with Internet Access: For research, simulations, and educational videos.
Optional: More Advanced Lab Equipment (depending on the level): Multimeter, oscilloscope, force sensors.

IV. Tips for Success:

Start with Observations and Questions: Encourage your child to notice the physics around them and ask questions.
Keep Experiments Hands-On and Engaging: Even complex concepts can be introduced through simple demonstrations.
Gradually Introduce Mathematical Formalism: Don't overwhelm younger children with equations.
Connect Physics to Other Subjects: Show how physics relates to math, chemistry, biology, and engineering.
Utilize Online Resources Wisely: Simulations, videos, and interactive websites can be very helpful, but choose reliable sources.
Don't Be Afraid to Say "I Don't Know": Use it as an opportunity to learn together.
Find a Good Physics Curriculum (Especially for High School): A structured curriculum can provide a roadmap and ensure comprehensive coverage.
Consider Unit Studies: Focus on a specific area of physics for a period (e.g., a unit on electricity, a unit on motion).
Visit Science Museums and Planetariums: These provide real-world examples and inspiration.
Encourage Participation in Science Fairs and Competitions: This can motivate learning and provide valuable experience.

V. Important Considerations:

Math Prerequisites: Ensure your child has the necessary math skills for the physics concepts being taught (especially algebra and calculus for higher levels).
Learning Styles: Adapt your teaching methods to your child's preferred learning style (visual, auditory, kinesthetic).
Time Commitment: Physics requires time for experimentation, problem-solving, and discussion.
Safety: Always prioritize safety during experiments. Supervise closely and teach safe practices.
Seeking Help: Don't hesitate to seek help from online forums, other homeschooling parents, or tutors if you encounter difficulties.
Keeping it Relevant: Connect physics to your child's interests (e.g., if they like cars, focus on mechanics; if they like music, explore sound waves).

By following this roadmap and keeping a focus on inquiry, experimentation, and real-world connections, you can provide your child with a strong foundation in physics at home. Remember to be flexible and enjoy the process of discovery!

A detailed course plan/roadmap for teaching mathematics at home

v like — Fri, 03 Jan 2025 03:21:59 GMT

That's fantastic you're taking the initiative to teach your child math at home! This can be a rewarding experience. Here's a detailed course plan/roadmap, broken down by age groups and key mathematical concepts. Remember that this is a guideline, and you'll need to adapt it based on your child's individual pace, interests, and learning style.

I. Foundational Principles (Apply to all ages):

Make it Fun and Engaging: Use games, stories, real-world examples, and hands-on activities to make learning enjoyable. Avoid turning it into a chore.
Focus on Understanding, Not Just Memorization: Encourage your child to explain their reasoning and not just recite facts.
Build on Previous Knowledge: Ensure they have a solid grasp of earlier concepts before moving on to more complex ones.
Use Manipulatives and Visual Aids: Tangible objects (like blocks, counters, and even food!) and visual representations (like drawings and diagrams) can make abstract concepts more concrete.
Regular Practice, But Keep it Short: Consistent, short sessions are more effective than long, infrequent ones.
Patience and Encouragement: Learning takes time. Be patient, offer positive reinforcement, and celebrate their progress.
Real-World Connections: Show how math is used in everyday life (e.g., measuring ingredients while cooking, counting money, telling time).
Embrace Mistakes as Learning Opportunities: Help your child understand that mistakes are a normal part of learning and a chance to identify areas for improvement.
Stay Flexible and Adapt: Be prepared to adjust your plan based on your child's progress and interests. Don't be afraid to slow down or revisit concepts as needed.

II. Course Roadmap by Age Group:

A. Preschool/Early Kindergarten (Ages 3-5): Focus: Number Sense and Early Spatial Reasoning

Key Concepts:
- Number Recognition (1-10 and beyond): Recognizing and naming numerals.
- Counting: Rote counting (saying numbers in order) and one-to-one correspondence (matching one object to one number).
- Subitizing: Recognizing small quantities (like 2 or 3) instantly without counting.
- Comparing Quantities: Understanding "more than," "less than," and "the same as."
- Shapes: Identifying and naming basic shapes (circle, square, triangle, rectangle).
- Spatial Reasoning: Understanding positional words (above, below, beside, in front of, behind).
- Patterns: Recognizing and creating simple patterns.
- Measurement (Informal): Comparing lengths, weights, and volumes using non-standard units (e.g., "This block is longer than that one").
Activities & Resources:
- Counting games: Counting fingers, toes, toys, snacks.
- Shape sorting and matching: Using shape sorters, drawing shapes.
- Building with blocks: Encouraging spatial reasoning and understanding of shapes.
- Singing counting songs and rhymes: Reinforces number sequences.
- Using manipulatives: Counting beans, buttons, or small toys.
- Reading counting books: Many engaging picture books introduce numbers.
- Simple pattern activities: Using beads, blocks, or stickers.
- Comparing activities: "Which pile has more?" "Are there enough cups for everyone?"

B. Kindergarten/Early Elementary (Ages 5-7): Focus: Building Number Sense and Basic Operations

Key Concepts:
- Number Recognition and Writing (up to 100 and beyond): Connecting numerals to their quantity.
- Counting (beyond 100): Understanding place value (tens and ones).
- Number Order and Sequencing: Identifying the number that comes before, after, or between.
- Addition and Subtraction (within 20): Understanding the concepts of joining and taking away.
- Introduction to Multiplication (as repeated addition): Using groups of objects.
- Measurement (Standard Units): Introducing units like inches, centimeters, and simple time telling (o'clock).
- Fractions (Basic): Understanding halves and wholes.
- Data and Graphing (Simple): Collecting and representing data using simple charts (e.g., bar graphs with pictures).
- Money (Introduction): Identifying coins and their values.
Activities & Resources:
- Manipulatives: Using base-ten blocks to understand place value.
- Number lines and hundreds charts: Visual aids for counting and understanding number relationships.
- Addition and subtraction games: Using dice, cards, or online games.
- Story problems: Presenting math problems in a narrative context.
- Measuring objects around the house: Using rulers and measuring tapes.
- Cutting objects into halves: Visualizing fractions.
- Creating simple graphs: Tracking favorite colors or pets.
- Playing store: Using play money to practice counting and basic transactions.
- Online math games and resources: (e.g., Khan Academy Kids, ABCmouse, Math Playground).

C. Upper Elementary (Ages 8-10): Focus: Developing Fluency with Operations and Introducing More Complex Concepts

Key Concepts:
- Multiplication and Division (Multi-digit): Developing fluency with multiplication facts and algorithms.
- Fractions (Operations): Adding, subtracting, multiplying, and dividing fractions.
- Decimals: Understanding decimal place value and performing basic operations.
- Measurement (Conversions): Converting between different units of measurement.
- Geometry (More Advanced Shapes and Angles): Identifying different types of triangles, quadrilaterals, and understanding angles.
- Area and Perimeter: Calculating the area and perimeter of simple shapes.
- Data Analysis and Probability (Introduction): Interpreting different types of graphs and understanding basic probability concepts.
- Algebraic Thinking (Introduction): Using variables to represent unknown quantities and solving simple equations.
Activities & Resources:
- Multiplication and division fact practice: Using flashcards, games, and online resources.
- Fraction manipulatives: Using fraction bars or circles.
- Real-world problems involving fractions and decimals: Cooking, measuring, and finances.
- Geometric construction activities: Using rulers, protractors, and compasses.
- Collecting and analyzing data: Conducting surveys and creating graphs.
- Solving simple algebraic puzzles: Using missing numbers or letter variables.
- Math workbooks and textbooks (appropriate for the grade level).
- Online resources: Khan Academy, IXL, Prodigy.

D. Middle School (Ages 11-13): Focus: Abstract Thinking and Formal Mathematical Reasoning

Key Concepts:
- Number Systems: Working with integers, rational numbers, and understanding the real number system.
- Algebra (Formal): Solving equations and inequalities, working with variables and expressions, graphing linear equations.
- Geometry (Congruence, Similarity, Transformations): Understanding geometric proofs and relationships.
- Ratios and Proportional Relationships: Solving problems involving proportions, scale factors, and percentages.
- Statistics and Probability (More Advanced): Calculating measures of central tendency, understanding probability distributions.
- Functions (Introduction): Understanding the concept of a function and its representation.
Activities & Resources:
- Algebra puzzles and word problems: Applying algebraic concepts to real-world situations.
- Geometric constructions and proofs: Developing logical reasoning skills.
- Working with scale drawings and maps: Applying proportional reasoning.
- Conducting statistical investigations: Collecting, analyzing, and interpreting data.
- Graphing calculators (optional, but helpful).
- Pre-algebra and algebra textbooks.
- Online resources: Khan Academy, Mathway, Wolfram Alpha.

E. High School (Ages 14-18): Focus: Advanced Mathematical Concepts and Preparation for Higher Education

This stage becomes more personalized based on the child's interests and future goals. Common subjects include:
- Algebra I & II: Advanced algebraic concepts, quadratic equations, polynomials, functions.
- Geometry (Advanced): Proofs, trigonometry, analytical geometry.
- Trigonometry: Study of triangles and trigonometric functions.
- Precalculus: Preparation for calculus, including advanced functions, logarithms, and sequences.
- Calculus: Differential and integral calculus.
- Statistics and Probability (Advanced): Inferential statistics, hypothesis testing.
- Discrete Mathematics: Logic, sets, combinatorics, graph theory.
Resources:
- High school math textbooks.
- Online courses: Khan Academy, Coursera, edX.
- Tutoring (if needed).
- Practice problems and standardized test preparation materials.

III. Essential Resources to Have On Hand:

Manipulatives: Base-ten blocks, counting bears, fraction circles/bars, pattern blocks, dice, playing cards.
Measuring Tools: Rulers, measuring tapes, scales, measuring cups and spoons.
Geometry Tools: Protractor, compass.
Writing Materials: Pencils, erasers, paper, notebooks, whiteboard/markers.
Visual Aids: Number lines, hundreds charts, geometric shape cutouts.
Workbooks and Textbooks (age-appropriate).
Online Resources and Apps (carefully selected and monitored).
Real-World Objects: Coins, food items, toys.

IV. Tips for Success:

Establish a Routine: Set aside dedicated time for math learning.
Create a Dedicated Learning Space: Minimize distractions.
Connect with Other Homeschooling Families: Share resources and support.
Stay Informed About Math Curriculum Standards: While you're not bound by public school standards, understanding them can be helpful.
Don't Be Afraid to Seek Help: Utilize online resources, tutors, or other homeschooling parents when you encounter challenges.
Celebrate Milestones: Acknowledge and celebrate your child's progress.
Continuously Assess Understanding: Don't just move on if your child struggles with a concept. Revisit and reinforce.

V. Important Considerations:

Learning Disabilities: If your child struggles significantly, consider exploring resources and strategies for learning disabilities in math (dyscalculia).
Keeping Up with Peers (if that's a concern): Compare your curriculum to established standards for your child's age group.
Socialization: Ensure your child has opportunities for social interaction outside of homeschooling.
Burnout (for both you and your child): Be flexible, take breaks, and keep learning enjoyable.

This is a comprehensive roadmap, and you can adjust it to fit your child's specific needs and your teaching style. Remember to enjoy the journey of learning together! Good luck!

Colors in Makie.jl

v like — Thu, 09 May 2024 06:52:30 GMT

# https://github.com/JuliaPlots/PlotUtils.jl/blob/master/src/colorschemes.jl
abstract type AbstractColorList end

abstract type ColorGradient <: AbstractColorList end

struct ContinuousColorGradient <: ColorGradient
    colors::ColorScheme
    values::Vector{Float64}

    function ContinuousColorGradient(colors, values = get_range(colors))
        c, v = prepare_continuous_cgrad_colors(colors, values)
        new(c, v)
    end
end

struct CategoricalColorGradient <: ColorGradient
    colors::ColorScheme
    values::Vector{Float64}

    function CategoricalColorGradient(colors, values = get_range(colors))
        c, v = prepare_categorical_cgrad_colors(colors, values)
        new(c, v)
    end
end

struct ColorPalette <: AbstractColorList
    colors::ColorScheme
end

"""
cgrad(colors, [values]; 
  categorical = nothing, scale = nothing, rev = false, alpha = nothing)

Construct a Colorgradient from `colors` and `values`.

`colors` can be a symbol for ColorSchemes.jl `ColorScheme`s, a `ColorScheme`, a vector of colors, a `ColorGradient` or a `ColorPalette`.
If `values` is an integer, it specifies the numbers of colors chosen equidistantly from the colorscheme specified by colors.
Otherwise vectors are accepted.
For continuous color gradients `values` indicate where between 0 and 1 the colors are positioned.
For categorical color gradients `values` indicate where a color ends and where a new one begins between 0 and 1.
0 and 1 are added to `values` if not already present.

If `rev` is `true` colors are reversed.
`scale` accepts the symbols `:log`, `:log10`, `:log2`, `:ln`, `:exp`, `:exp10` or functions.
If `alpha` is set, it is applied to all colors.
"""


"""
palette(colors, [n]; rev = false, alpha = nothing)

Construct a `ColorPalette`.
Accepts symbols for Colorschemes.jl `ColorScheme`s, `ColorScheme`s, vectors of colors, `ColorGradient`s and `ColorPalettes`.
If 'n' is provided, `n` colors are chosen equidistantly from the colorscheme.
If `rev` is `true` colors are reversed.
"""

# https://juliagraphics.github.io/ColorSchemes.jl/stable/
] add ColorSchemes

using ColorSchemes

ColorSchemes.Purples_5 
# => a ColorScheme 

colorschemes[:Purples_5]
# => a ColorScheme 

ColorSchemes.Purples_5.colors
# => array of five RGB colors

ColorSchemes.Purples_5.colors[3]
# => the third color in the colorscheme

get(ColorSchemes.Purples_5, 0.5)
# => the midway point of the colorscheme

colorschemes
# => Dict{Symbol, ColorScheme} with 983 entries

findcolorscheme("purple")
# => display list of matching schemes

# https://juliagraphics.github.io/ColorSchemeTools.jl/stable/makingschemes/
roygbiv = [
    colorant"red",
    colorant"orange",
    colorant"yellow",
    colorant"green",
    colorant"blue",
    colorant"indigo",
    colorant"violet"
]
scheme = make_colorscheme(roygbiv, 10)

# You can supply the colors in any format, as long as it's a Colorant:
cols = Any[
    RGB(0, 0, 1),
    Gray(0.5),
    HSV(50., 0.7, 1.),
    Gray(0.4),
    LCHab(54, 105, 40),
    HSV(285., 0.9, 0.8),
    colorant"#FFEEFF",
    colorant"hotpink",
]
scheme = make_colorscheme(cols, 8)

Color attributes

color=theme(scene, :linecolor) sets the color of the line. If no color is set, multiple calls to line! will cycle through the axis color palette. Otherwise, one can set one color per line point by passing a Vector{<:Colorant}, or one colorant for the whole line. If color is a vector of numbers, the colormap args are used to map the numbers to colors.

cycle::Vector{Symbol} = [:color] sets which attributes to cycle when creating multiple plots.

colormap::Union{Symbol, Vector{<:Colorant}} = :viridis sets the colormap that is sampled for numeric colors. PlotUtils.cgrad(...), Makie.Reverse(any_colormap) can be used as well, or any symbol from ColorBrewer or PlotUtils. To see all available color gradients, you can call Makie.available_gradients().

colorscale::Function = identity color transform function. Can be any function, but only works well together with Colorbar for identity, log, log2, log10, sqrt, logit, Makie.pseudolog10 and Makie.Symlog10.

colorrange::Tuple{<:Real, <:Real} sets the values representing the start and end points of colormap.

nan_color::Union{Symbol, <:Colorant} = RGBAf(0,0,0,0) sets a replacement color for color = NaN.

lowclip::Union{Nothing, Symbol, <:Colorant} = nothing sets a color for any value below the colorrange.

highclip::Union{Nothing, Symbol, <:Colorant} = nothing sets a color for any value above the colorrange.

alpha = 1.0 sets the alpha value of the colormap or color attribute. Multiple alphas like in plot(alpha=0.2, color=(:red, 0.5), will get multiplied.

How to use the CDS API with curl

v like — Tue, 16 Apr 2024 15:21:52 GMT

The Climate Data Store (CDS) Application Program Interface (API) is a service providing programmatic access to CDS data like ERA5.

ERA5 is the fifth generation ECMWF reanalysis for the global climate and weather for the past 8 decades. Data is available from 1940 onwards. ERA5 replaces the ERA-Interim reanalysis.

The CDS has provided a python based API client library cdsapi. It provides support for both Python 2.7.x and Python 3.

Python cdsapi single levels example

code from https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=form

import cdsapi

c = cdsapi.Client()

c.retrieve(
    'reanalysis-era5-single-levels',
    {
        'product_type': 'reanalysis',
        'format': 'grib',
        'variable': [
            '2m_dewpoint_temperature', '2m_temperature', 'convective_available_potential_energy',
            'convective_precipitation', 'large_scale_precipitation', 'mean_sea_level_pressure',
            'total_precipitation',
        ],
        'year': '2024',
        'month': [
            '03', '04',
        ],
        'day': [
            '01', '02', '30',
            '31',
        ],
        'time': [
            '00:00', '01:00', '02:00',
            '03:00', '04:00', '05:00',
            '06:00', '07:00', '08:00',
            '09:00', '10:00', '11:00',
            '12:00', '13:00', '14:00',
            '15:00', '16:00', '17:00',
            '18:00', '19:00', '20:00',
            '21:00', '22:00', '23:00',
        ],
        'area': [
            50, 90, 20,
            120,
        ],
    },
    'download.grib')

Python cdsapi pressure levels example

code from https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels?tab=form

import cdsapi

c = cdsapi.Client()

c.retrieve(
    'reanalysis-era5-pressure-levels',
    {
        'product_type': 'reanalysis',
        'format': 'grib',
        'pressure_level': [
            '500', '700',
        ],
        'year': '2024',
        'month': [
            '03', '04',
        ],
        'day': [
            '01', '02', '30',
            '31',
        ],
        'time': [
            '00:00', '01:00', '02:00',
            '03:00', '04:00', '05:00',
            '06:00', '07:00', '08:00',
            '09:00', '10:00', '11:00',
            '12:00', '13:00', '14:00',
            '15:00', '16:00', '17:00',
            '18:00', '19:00', '20:00',
            '21:00', '22:00', '23:00',
        ],
        'area': [
            50, 90, 20,
            120,
        ],
        'variable': [
            'divergence', 'geopotential',
        ],
    },
    'download.grib')

We can also use the CDS API without Python. Here I will show you how to use the CDS API with curl to download ERA5 data.

Check API status

curl https://cds.climate.copernicus.eu/api/v2/status.json

{
    "info": ["Welcome to the CDS"],
    "warning": []
}

Authorization Required

curl -X POST -H "Content-Type: application/json" 
    https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels


401 Authorization Required

401 Authorization Required
nginx/1.20.2

If you don't have an account, please self register at the CDS registration page and go to the steps below.

If you are not logged, please login and go to the step below.

Copy the key displayed at https://cds.climate.copernicus.eu/api-how-to

Agreed to the required terms and conditions

curl -X POST -H "Content-Type: application/json" 
    -u 123456:2d0b6cac-e6cb-4d1b-85df-1234560f4dd6 
    -d '{ "variable": "temperature", "pressure_level": "1000", "product_type": "reanalysis", "date": "2017-12-01/2017-12-02", "time": "12:00", "format": "grib" }' 
    https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels

{
  "message": "Client has not agreed to the required terms and conditions.",
  "url": "http://copernicus-climate.eu/exc/missing-terms",
  "reason": "Client has not agreed to the required terms and conditions.",
  "context": {
    "required_terms": [{
      "title": "Licence to use Copernicus Products",
      "url": "https://cds.climate.copernicus.eu/cdsapp/#!/terms/licence-to-use-copernicus-products",
      "reason": "missing"
    }]
  },
  "permanent": false,
  "who": "client"
}

To access this resource, you first need to accept the terms of "Licence to use Copernicus Products" at https://cds.climate.copernicus.eu/cdsapp/#!/terms/licence-to-use-copernicus-products

Request will queued

curl -X POST -H "Content-Type: application/json" 
    -u 123456:2d0b6cac-e6cb-4d1b-85df-1234560f4dd6 
    -d '{ "variable": "temperature", "pressure_level": "1000", "product_type": "reanalysis", "date": "2017-12-01/2017-12-02", "time": "12:00", "format": "grib" }' 
    https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels

{
  "state": "queued",
  "request_id": "baa0698c-b035-4ace-a463-a3ba4639df69",
  "specific_metadata_json": {
    "top_request_origin": "api"
  }
}

Check request_id state

curl -u 123456:2d0b6cac-e6cb-4d1b-85df-1234560f4dd6 
    https://cds.climate.copernicus.eu/api/v2/tasks/baa0698c-b035-4ace-a463-a3ba4639df69

{
  "state": "completed",
  "request_id": "baa0698c-b035-4ace-a463-a3ba4639df69",
  "location": "https://download-0001-clone.copernicus-climate.eu/cache-compute-0001/cache/data6/adaptor.mars.internal-1713276220.7536583-952-1-baa0698c-b035-4ace-a463-a3ba4639df69.grib",
  "content_length": 4153200,
  "content_type": "application/x-grib",
  "sent_to_rmq_at": "2024-04-16T14:03:40.448Z",
  "specific_metadata_json": {
    "top_request_origin": "api"
  }
}

Download the file at location

curl -o era5-test.grib https://download-0001-clone.copernicus-climate.eu/cache-compute-0001/cache/data6/adaptor.mars.inter
nal-1713276220.7536583-952-1-baa0698c-b035-4ace-a463-a3ba4639df69.grib

That's all!

Julia Notes Day5 -- REPL help/pkg/search mode and Depot

v like — Sat, 23 Mar 2024 10:17:58 GMT

Install Julia

All official Julia binaries produce portable installations. Once installed, the directory in which Julia was installed can be moved to a different location on the same computer, or even to a different computer.

wget https://julialang-s3.julialang.org/bin/linux/x64/1.10/julia-1.10.2-linux-x86_64.tar.gz
tar zxvf julia-1.10.2-linux-x86_64.tar.gz

The generic Linux and FreeBSD binaries do not require any special installation steps, but you will need to ensure that your system can find the julia executable. The directory where Julia is installed is referred to as .

To run Julia, you can do any of the following:

Invoke the julia executable by using its full path: /bin/julia
Create a symbolic link to julia inside a folder which is on your system PATH
Add Julia's bin folder (with full path) to your system PATH environment variable

To add Julia's bin folder (with full path) to PATH environment variable, you can edit the ~/.bashrc (or ~/.bash_profile) file. Open the file in your favourite editor and add a new line as follows:

export PATH="$PATH:/workspace/empty/julia-1.10.2/bin"

REPL

The easiest way to learn and experiment with Julia is by starting an interactive session (also known as a read-eval-print loop or "REPL") by double-clicking the Julia executable or running julia from the command line:

$ julia

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.2 (2024-03-01)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |


julia> 1 + 2
3

julia> ans
3

julian mode

The REPL has five main modes of operation. The first and most common is the Julian prompt. It is the default mode of operation; each new line initially starts with julia>. It is here that you can enter Julia expressions. Hitting return or enter after a complete expression has been entered will evaluate the entry and show the result of the last expression.

help mode

You can use the REPL as a learning resource by switching into the help mode. Switch to help mode by pressing ? at an empty julia> prompt, before typing anything else. Typing a keyword in help mode will fetch the documentation for it, along with examples.

julia> using UnicodePlots

help?> lineplot
search: lineplot lineplot!

  lineplot(; kw...)
  lineplot(y; kw...)
  lineplot(x, y; kw...)
  lineplot!(p, args...; kw...)

  Description
  ≡≡≡≡≡≡≡≡≡≡≡

  Draws a path through the given points on a new canvas.

  The first (optional) vector x should contain the horizontal positions for all the points along the path. The second
  vector y should then contain the corresponding vertical positions respectively. This means that the two vectors must be
  of the same length and ordering.

  Usage
  ≡≡≡≡≡

  lineplot([x], y; head_tail = nothing, ..., name = "")
  lineplot([start], [stop], fun; kw...)

  Arguments
  ≡≡≡≡≡≡≡≡≡

    •  fun : a unary function f: R -> R that should be evaluated, and drawn as a path from start to stop (numbers in
       the domain).

    •  head_tail : color the line head and/or tail with the complement of the chosen color (:head, :tail, :both).

    •  head_tail_frac : fraction of the arrow head or tail (e.g. provide 0.1 for 10%).

    •  x : horizontal position for each point (can be a real number or of type TimeType), if omitted, the axes of y
       will be used as x.

    •  format : specify the ticks date format (TimeType only).

    •  color::Symbol = :auto : Vector of colors, or scalar - choose from (:green, :blue, :red, :yellow, :cyan,
       :magenta, :white, :normal, :auto), use an integer in [0-255], or provide 3 integers as RGB components.

    •  y : vertical position for each point.

    •  title::String = "" : text displayed on top of the plot.

    •  name::String = "" : current drawing annotation displayed on the right.

    •  xlabel::String = "" : text displayed on the x axis of the plot.

    •  ylabel::String = "" : text displayed on the y axis of the plot.

    •  xscale::Symbol = :identity : x-axis scale (:identity, :ln, :log2, :log10), or scale function e.g. x ->
       log10(x).

    •  yscale::Symbol = :identity : y-axis scale.

    •  labels::Bool = true : show plot labels.

    •  border::Symbol = :solid : plot bounding box style (:corners, :solid, :bold, :dashed, :dotted, :ascii, :none).

    •  margin::Int = 3 : number of empty characters to the left of the whole plot.

    •  padding::Int = 1 : left and right space between the labels and the canvas.

    •  height::Int = 15 : number of canvas rows, or :auto.

    •  width::Int = 40 : number of characters per canvas row, or :auto.

    •  xlim::Tuple = (0, 0) : plotting range for the x axis ((0, 0) stands for automatic).

    •  ylim::Tuple = (0, 0) : plotting range for the y axis.

    •  xflip::Bool = false : set true to flip the x axis.

    •  yflip::Bool = false : set true to flip the y axis.

    •  canvas::UnionAll = UnicodePlots.BrailleCanvas : type of canvas used for drawing.

    •  grid::Bool = true : draws grid-lines at the origin.

    •  compact::Bool = false : compact plot labels.

    •  unicode_exponent::Bool = true : use Unicode symbols for exponents: e.g. 10²⸱¹ instead of 10^2.1.

    •  blend::Bool = true : blend colors on the underlying canvas.

  Author(s)
  ≡≡≡≡≡≡≡≡≡

    •  Christof Stocker (github.com/Evizero)

    •  Milktrader (github.com/milktrader)

  Examples
  ≡≡≡≡≡≡≡≡

  julia> lineplot([1, 2, 7], [9, -6, 8]; title = "My Lineplot")
         ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀My Lineplot⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 
         ┌────────────────────────────────────────────────┐ 
      10 │⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ ⠀⠀ ⠀⠀⠀│ 
         │⢇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡠│ 
         │⠘⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠤⠊⠁⠀│ 
         │⠀⢣⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⠔⠊⠁⠀⠀⠀⠀│ 
         │⠀⠈⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⠔⠊⠀⠀⠀⠀⠀⠀⠀⠀│ 
         │⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡠⠔⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ 
         │⠀⠀⠀⢇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡠⠒⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ 
         │⠤⠤⠤⠼⡤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⢤⠤⠶⠥⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤│ 
         │⠀⠀⠀⠀⢣⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠤⠊⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ 
         │⠀⠀⠀⠀⠈⡆⠀⠀⠀⠀⠀⠀⠀⣀⠔⠊⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ 
         │⠀⠀⠀⠀⠀⢱⠀⠀⠀⠀⡠⠔⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ 
         │⠀⠀⠀⠀⠀⠀⢇⡠⠔⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀│ 
         │⠀⠀⠀⠀⠀⠀⠈⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀│ 
         │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠀⠀│ 
     -10 │⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀⠀│ 
         └────────────────────────────────────────────────┘ 
         ⠀1⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀7⠀ 

  See also
  ≡≡≡≡≡≡≡≡

  Plot, scatterplot, stairs, BrailleCanvas, BlockCanvas, AsciiCanvas, DotCanvas

pkg mode

The Package manager mode accepts specialized commands for loading and updating packages. It is entered by pressing the ] key at the Julian REPL prompt and exited by pressing CTRL-C or pressing the backspace key at the beginning of the line. The prompt for this mode is pkg>. It supports its own help-mode, which is entered by pressing ? at the beginning of the line of the pkg> prompt. The Package manager mode is documented in the Pkg manual, available at https://julialang.github.io/Pkg.jl/v1/.

shell mode

Just as help mode is useful for quick access to documentation, another common task is to use the system shell to execute system commands. Just as ? entered help mode when at the beginning of the line, a semicolon (;) will enter the shell mode. And it can be exited by pressing backspace at the beginning of the line.

search mode

In all of the above modes, the executed lines get saved to a history file, which can be searched. To initiate an incremental search through the previous history, type ^R – the control key together with the r key. The prompt will change to (reverse-i-search)`':, and as you type the search query will appear in the quotes. The most recent result that matches the query will dynamically update to the right of the colon as more is typed. To find an older result using the same query, simply type ^R again.

Just as ^R is a reverse search, ^S is a forward search, with the prompt (i-search)`':. The two may be used in conjunction with each other to move through the previous or next matching results, respectively.

Depot Directory

The JULIA_DEPOT_PATH environment variable is used to populate the global Julia DEPOT_PATH variable, which controls where the package manager, as well as Julia's code loading mechanisms, look for package registries, installed packages, named environments, repo clones, cached compiled package images, configuration files, and the default location of the REPL's history file.

Here is an overview of some of the subdirectories that may exist in a depot:

compiled: Contains precompiled *.ji files for packages. Maintained by Julia.
environments: Default package environments. For instance the global environment for a specific julia version. Maintained by Pkg.jl.
logs: Contains logs of Pkg and REPL operations. Maintained by Pkg.jl and Julia.
packages: Contains packages, some of which were explicitly installed and some which are implicit dependencies. Maintained by Pkg.jl.
registries: Contains package registries. By default only General. Maintained by Pkg.jl.
scratchspaces: Contains content that a package itself installs via the Scratch.jl package. Pkg.gc() will delete content that is known to be unused.

export JULIA_DEPOT_PATH="/workspace/empty/depot"

Julia Notes Day4 -- Module,Package and Environment

v like — Tue, 19 Mar 2024 09:55:51 GMT

Module

Modules in Julia help organize code into coherent units. They are delimited syntactically inside module NameOfModule ... end.Typically, in larger Julia packages you will see module code organized into files, eg

module SomeModule

# export, using, import statements are usually here; we discuss these below

include("file1.jl")
include("file2.jl")

end

Files and file names are mostly unrelated to modules; modules are associated only with module expressions. One can have multiple files per module, and multiple modules per file. include behaves as if the contents of the source file were evaluated in the global scope of the including module.

# math.jl
module SomeModuleMath
export add

add(x, y=5) = x + y
end

To load a module from a package, the statement using ModuleName can be used. To load a module from a locally defined module, a dot needs to be added before the module name like using .ModuleName.

# main1.jl
include("math.jl") # first include file for locally defined module
using .SomeModuleMath

println(add(3,3))

In contrast, import brings only the module name into scope.

An identifier brought into scope by import or using can be renamed with the keyword as. This is useful for working around name conflicts as well as for shortening names.

# main2.jl
include("math.jl") # first include file for locally defined module
import .SomeModuleMath as MyMath

println(MyMath.add(5,3))

Package

Pkg is Julia's package manager.

Pkg comes with a REPL. Enter the Pkg REPL by pressing ] from the Julia REPL. To get back to the Julia REPL, press Ctrl+C or backspace (when the REPL cursor is at the beginning of the input).

To add a package, use add:

(@v1.10) pkg> add UnicodePlots

After the package is installed, it can be loaded into the Julia session:

julia> using UnicodePlots

julia> lineplot(-π/2, 2π, [cos, sin])

The status command (or the shorter st command) can be used to see installed packages.

To remove packages, use rm (or remove).

Use up (or update) to update the installed packages.

If a package is not in a registry, it can be added by specifying a URL to the Git repository:

(@v1.10) pkg> add https://github.com/fredrikekre/ImportMacros.jl

Environment

You may have noticed the (@v1.10) in the REPL prompt. This lets us know that v1.10 is the active environment. Different environments can have different totally different packages and versions installed from another environment. The active environment is the environment that will be modified by Pkg commands such as add, rm and update.

So far we have added packages to the default environment at ~/.julia/environments/v1.10. It is however easy to create other, independent, projects. This approach has the benefit of allowing you to check in a Project.toml, and even a Manifest.toml if you wish, into version control (e.g. git) alongside your code.

active environment

In order to create a new project, create a directory for it and then activate that directory to make it the "active project", which package operations manipulate.

To set the active environment, use activate:

(@v1.10) pkg> activate tutorial
[ Info: activating new environment at `~/tutorial/Project.toml`.

(tutorial) pkg>

We can ask for information about the active environment by using status.

It should be pointed out that when two projects use the same package at the same version, the content of this package is not duplicated.

If the project contains a manifest, this will install the packages in the same state that is given by that manifest. Otherwise, it will resolve the latest versions of the dependencies compatible with the project.

instantiate

Note that activate by itself does not install missing dependencies. If you only have a Project.toml, a Manifest.toml must be generated by "resolving" the environment, then any missing packages must be installed and precompiled. instantiate does all this for you.

If you already have a resolved Manifest.toml, then you will still need to ensure that the packages are installed and with the correct versions. Again instantiate does this for you.

In short, instantiate is your friend to make sure an environment is ready to use. If there's nothing to do, instantiate does nothing.

shell> git clone https://github.com/JuliaLang/Example.jl.git
Cloning into 'Example.jl'...
...

(@v1.10) pkg> activate Example.jl
Activating project at `~/Example.jl`

(Example) pkg> instantiate
  No Changes to `~/Example.jl/Project.toml`
  No Changes to `~/Example.jl/Manifest.toml`

shared environment

A "shared" environment is simply an environment that exists in ~/.julia/environments. The default v1.10 environment is therefore a shared environment.

Shared environments can be activated with the --shared flag to activate:

(@v1.10) pkg> activate --shared mysharedenv
  Activating project at `~/.julia/environments/mysharedenv`

(@mysharedenv) pkg>

precompile package

By default, any package that is added to a project or updated in a Pkg action will be automatically precompiled, along with its dependencies.

How AI weather models like ECMWF-AIFS and GraphCast work

v like — Tue, 19 Mar 2024 08:58:55 GMT

Original article: https://twitter.com/Brady_Wx/status/1769910256354627891

GraphCast: Learning skillful medium-range global weather forecasting

I see a lot of people posting false information about AI weather models like ECMWF-AIFS and GraphCast. This is brand new technology, so misunderstandings are to be expected. I'll try to explain how they work in way that is simplified and easy to understand. This is a bit over-simplified, but should be sufficient to help weather enthusiasts, storm chasers, etc understand how these models work to an extent.

These models aren't weather models designed by AI. They aren't ChatGPT wrappers. In fact, they aren't even related to that "kind" of AI (Large Language Models). These models use a somewhat similar underlying 'architecture', but do not use human-like logic in any way. "Machine-Learning" is probably an easier way to understand these models than the term "AI".

So, briefly, I will describe how neural networks work in general.

Problem

Imagine you are trying to predict whether or not a severe thunderstorm is likely at a precise location (lets say Amarillo Tx), and you want to build an extremely basic neural network for this task. This basic neural network is called a multi-layer perceptrton (MLP).

Features

Some features to consider at this location would be dew-point, srh, wind shear, cape.. we'll leave it at those 4 for this example. So our MLP will be taking the DP, srh, 500mb wind shear, and CAPE at this location and it will be outputting the probability of a severe thunderstorm. The model will need to somehow learn what combinations of DP,srh,ws, and cape generally mean severe thunderstorms and what combinations do not.

Let's say we've collected 100 data points for Amarillo Texas (so 100 days of data). In 50 of those, a severe thunderstorm happened, and in the other 50 there weren't any severe thunderstorms. So what we now have is 100 data points where each data point contains the DP, srh, 500mb wind shear, and CAPE in Amarillo on that day, as well as whether or not a severe thunderstorm occurred. So we have our training data and we are ready to feed into MLP.

Model hidden layers

The MLP will go datapoint-by-datapoint, for each datapoint the data will be input and will enter the first "hidden layer". In this hidden layer, the model applies some random multiplication (weights) and addition (bias) to each input, n number of times (the number of "neurons" in this hidden layer). Each neuron is then connected to the neurons in the next hidden layer. The number of hidden layers is the "depth" of the model and is customizable by the model engineer. More complex models require more neurons and more depth, to compensate for the complexity of the issue at hand.

In each hidden layer, the neurons from the previous layer are fully connected to the neurons of the current layer, weights (multiply) and biases (add) are applied, and then they move to the next layer. A special function is applied between layers to ensure that the model is able to learn non-linearity (not really relevant to a simple discussion here). When the data arrives at the final/output layer, the biases/weights from the final hidden layer are all congealed into a single neuron. A special function is applied here which adjusts all of the data into a single value between 0 and 1, and that is your probability.

Training

The model will begin training with random weights and biases but with each datapoint will propagate backwards to try to "figure out" how to adjust the weights and biases to reduce the error. After many datapoints and epochs (iterations over the same data), the model (if it is well designed with quality, diverse data) will be able to be applied to new data (only inputs) and output realistic probabilities (outputs).

So how exactly do these new AI/ML models work? Both AIFS and GraphCast use an architecture called encoder-processor-decoder Graph Neural Networks. This means that the model takes the input data and puts it through 3 steps (encoder, processor, decoder). I will mostly focus on GraphCast as it is open source.

Encoder gridpoints

In the first step, the encoder, the models take initial conditions (various meteorological fields such as temperature, geopotential heights, humidity, and more) across the globe and "encode" these datapoints onto a mesh grid (a grid where gridpoints are equidistant from one another) and connect each gridpoint to other gridpoints that are nearby as well as far away. This mapping is done by a neural network, which just means that the model is trained on how to make these connections most efficiently. Each gridpoint is now considered a "node" in the graph with "edges" connecting them to gridpoints near and far (7 levels of resolution reduction allow for connections near and far (high resolution and low resolution)).

Forward-propagating

"Connected" in this context just means that the program is informing the model that the gridpoints are connected and how they are connected. So the gridpoints (and their data) are connected, now what's next? The model enters the "processor" stage. In the processor stage, the model passes data between connected gridpoints 16 times. When a node (or gridpoint) receives information from a connected node, it updates its values and pushes data to the connected gridpoints. This message (or data) exchange occurs 16 times. The changes that each gridpoint applies is dependent on the data it is receiving from neighboring gridpoints, and this is entirely dependent on the training.

Back-propagating and Decoder gridpoints

The final step is the "decoder" stage. In this stage, the data is plotted from the abstract processing stage, where it is split between various grid resolutions, back to the original 0.25deg lat-lon grid. This mapping is done through a single-layered graph neural network (the model simply learns how to map the data back to the original lat-lon grid). In training, the model compares the output to the expected output (target data based on past analysis data).. if the difference is huge, the model goes backwards through the steps and tries to adjust the values so that the error is reduced. It does this over and over again for each past event... and then goes over all of the events again likely dozens to hundreds of times. This allows the model to learn all of these various "trainable" factors (connections between gridpoints, how to change the data based on neighboring data, how to plot the data back to the original latlon grid, etc)..

Chained forecast

This is only for a 6hr forecast. The outputs from the 6hr forecast are then fed back into the model as inputs, producing a fresh 6hr forecast (total of 12hrs).. these forecasts are continuously chained together into the future indefinitely.. usually 240hrs currently. Eventually, the data becomes oversmooth and unrealistic.

So these models are very complex... a lot of that is simplified. If you'd like a deeper dive into this, I suggest reading the full paper on GraphCast published by google deepmind.

Test

Btw: these models have been extensively tested with published verification data. They score better on every tested metric compared to ECMWF, GFS, and every other physics model. This includes with severe weather such as tropical cyclones.

Julia Notes Day3 -- Composite and Parametric Types

v like — Fri, 15 Mar 2024 03:59:46 GMT

Composite types are called records, structs, or objects in various languages.

struct Foo
    bar
    baz::Int
    qux::Float64
end

foo = Foo("Hello, world.", 23, 1.5) # Foo("Hello, world.", 23, 1.5)

foo.bar # "Hello, world."

Composite objects declared with struct are immutable; they cannot be modified after construction.

An immutable object might contain mutable objects, such as arrays, as fields. Those contained objects will remain mutable; only the fields of the immutable object itself cannot be changed to point to different objects.

Where required, mutable composite objects can be declared with the keyword mutable struct.

In cases where one or more fields of an otherwise mutable struct is known to be immutable, one can declare these fields as such using const as shown below.

mutable struct Baz
    a::Int
    const b::Float64
end

baz = Baz(1, 1.5) # Baz(1, 1.5)
baz.a = 3;
baz # Baz(3, 1.5)

An important and powerful feature of Julia's type system is that it is parametric: types can take parameters, so that type declarations actually introduce a whole family of new types – one for each possible combination of parameter values.

struct Point{T<:Real}
    x::T
    y::T
end

Without any explicitly provided inner constructors, the declaration of the composite type Point{T<:Real} automatically provides an inner constructor, Point{T}, for each possible type T<:Real, that behaves just like non-parametric default inner constructors do. It also provides a single general outer Point constructor that takes pairs of real arguments, which must be of the same type. This automatic provision of constructors is equivalent to the following explicit declaration:

struct Point{T<:Real}
   x::T
   y::T
   Point{T}(x,y) where {T<:Real} = new(x,y)
end

Point(x::T, y::T) where {T<:Real} = Point{T}(x,y);

By default, instances of parametric composite types can be constructed either with explicitly given type parameters or with type parameters implied by the types of the arguments given to the constructor. Here are some examples:

p1 = Point(7,5) ## implicit T ## Point{Int64}(7,5)
p2 = Point(2,3) ## implicit T ## Point{Int64}(2,3)

Point{Float64}(1,2) ## explicit T ## Point{Float64}(1.0, 2.0)

p1 = Point(5,1.5) 
# ERROR: MethodError: no method matching Point(::Int64, ::Float64)

For a more general way to make all such calls work sensibly, all it takes is the following outer method definition to make all calls to the general Point constructor work as one would expect:

Point(x::Real, y::Real) = Point(promote(x,y)...);

# Point(Float64, Int) -> Point(Float64, Float64)
Point(1.5,2) # Point{Float64}(1.5, 2.0)

The promote function converts all its arguments to a common type – in this case Float64. With this method definition, the Point constructor promotes its arguments the same way that numeric operators like + do, and works for all kinds of real numbers.

In mainstream object oriented languages, such as C++, Java, Python and Ruby, composite types also have named functions associated with them, and the combination is called an "object".

In Julia, all values are objects, but functions are always not bundled with the objects they operate on.This is necessary since Julia chooses which method of a function to use by multiple dispatch, meaning that the types of all of a function's arguments are considered when selecting a method, rather than just the first one.

Base.show(io::IO, z::Point) = print(io, "{\n  x=$(z.x)\n  y=$(z.y)\n}")

Point(2.3, 5)
#= 
{
  x=2.3
  y=5.0
}
=#

Julia Notes Day2 -- string and collections

v like — Wed, 13 Mar 2024 10:53:53 GMT

Char and String

A Char value represents a single character

c = 'x'
d = Int('x') # 120
e = Char(120)  # 'x'
f = '\u2200' # '∀': Unicode U+2200 (category Sm: Symbol, math)
g = 'A' + 1 # 'B'

String literals are delimited by double quotes or triple double quotes (not single quotes).

s = "\u2200 x \u2203 y" # "∀ x ∃ y"
s *= "世界" # "∀ x ∃ y世界"
length(s) # 9
sizeof(s) # 17
s[end] # '界': Unicode U+754C (category Lo: Letter, other)

findfirst('界',s) # 15

# Sequential iteration is implemented by the iterate function. 
# The general for loop:
for c in s
    print(c*' ')
end
# ∀   x   ∃   y 世 界

String literals are encoded using the UTF-8 encoding. UTF-8 is a variable-width encoding, meaning that not all characters are encoded in the same number of bytes ("code units"). In UTF-8, ASCII characters — i.e. those with code points less than 0x80 (128) – are encoded as they are in ASCII, using a single byte, while code points 0x80 and above are encoded using multiple bytes — up to four per character.

String indices in Julia refer to code units (= bytes for UTF-8), the fixed-width building blocks that are used to encode arbitrary characters (code points). This means that not every index into a String is necessarily a valid index for a character. If you index into a string at such an invalid byte index, an error is thrown

greet = "Hello"; whom = "world";

#Julia provides * for string concatenation
greet * ' ' * whom # "Hello world"

#interpolation into string literals using $
"$greet, $whom." # "Hello, world."

# triple-quoted strings are dedented to the level of the least-indented line.
str = """
           Hello,
           world.
         """
# "  Hello,\n  world.\n"

"1 + 2 = 3" == "1 + 2 = $(1 + 2)" # true

# repeat("A", 10)
'A'^10 # "AAAAAAAAAA"
"B"^10 # "BBBBBBBBBB"
"BA"^10 # "BABABABABABABABABABA"

Range and Vector

V = [i/2 for i in 1:10]

# Comprehensions can also be written without the enclosing square brackets, 
# producing an object known as a generator.
sum(1/n^2 for n=1:1000)

# convert a range to a vector by using the collect function.
vec = collect(1:2:7)
println(vec)  # Output: [1, 3, 5, 7]

Matrix

A = [1 2; 3 4] 
#= 2×2 Matrix{Int64}:
 1  2
 3  4
=#
B = A*A*A
#= 2×2 Matrix{Int64}:
 37   54
 81  118
=#
B == A^3 # true

C = A * 3
C == 3A  # true

sin.(A)
#= 2×2 Matrix{Float64}:
 0.841471   0.909297
 0.14112   -0.756802
=#

sin(A)
#= 2×2 Matrix{Float64}:
 -0.465581  -0.148424
 -0.222637  -0.688218
=#

A*[1,2]
#= 2-element Vector{Int64}:
  5
 11
=#

Dict

Dict("A"=>1, "B"=>2)
#= Dict{String, Int64} with 2 entries:
  "B" => 2
  "A" => 1
=#

List of functions of Collections

Base.iterate — Function

Base.IteratorSize — Type

Base.IteratorEltype — Type

Base.AbstractRange — Type

Base.OrdinalRange — Type

Base.AbstractUnitRange — Type

Base.StepRange — Type

Base.UnitRange — Type

Base.LinRange — Type

Base.isempty — Function

Base.empty! — Function

Base.length — Function

Base.checked_length — Function

Base.in — Function

Base.:∉ — Function

Base.eltype — Function

Base.indexin — Function

Base.unique — Function

Base.unique! — Function

Base.allunique — Function

Base.allequal — Function

Base.reduce — Method

Base.reduce — Method

Base.foldl — Method

Base.foldr — Method

Base.maximum — Function

Base.maximum! — Function

Base.minimum — Function

Base.minimum! — Function

Base.extrema — Function

Base.extrema! — Function

Base.argmax — Function

Base.argmin — Function

Base.findmax — Function

Base.findmin — Function

Base.findmax! — Function

Base.findmin! — Function

Base.sum — Function

Base.sum! — Function

Base.prod — Function

Base.prod! — Function

Base.any — Method

Base.any — Method

Base.any! — Function

Base.all — Method

Base.all — Method

Base.all! — Function

Base.count — Function

Base.foreach — Function

Base.map — Function

Base.map! — Function

Base.mapreduce — Method

Base.mapfoldl — Method

Base.mapfoldr — Method

Base.first — Function

Base.last — Function

Base.front — Function

Base.tail — Function

Base.step — Function

Base.collect — Method

Base.collect — Method

Base.filter — Function

Base.filter! — Function

Base.replace — Method

Base.replace — Method

Base.replace! — Function

Base.rest — Function

Base.split_rest — Function

Base.getindex — Function

Base.setindex! — Function

Base.firstindex — Function

Base.lastindex — Function

Base.AbstractDict — Type

Base.Dict — Type

Base.IdDict — Type

Base.WeakKeyDict — Type

Base.ImmutableDict — Type

Base.haskey — Function

Base.get — Function

Base.get! — Function

Base.getkey — Function

Base.delete! — Function

Base.pop! — Method

Base.keys — Function

Base.values — Function

Base.pairs — Function

Base.merge — Function

Base.mergewith — Function

Base.merge! — Function

Base.mergewith! — Function

Base.sizehint! — Function

Base.keytype — Function

Base.valtype — Function

Base.AbstractSet — Type

Base.Set — Type

Base.BitSet — Type

Base.union — Function

Base.union! — Function

Base.intersect — Function

Base.setdiff — Function

Base.setdiff! — Function

Base.symdiff — Function

Base.symdiff! — Function

Base.intersect! — Function

Base.issubset — Function

Base.:⊈ — Function

Base.:⊊ — Function

Base.issetequal — Function

Base.isdisjoint — Function

Base.push! — Function

Base.pop! — Function

Base.popat! — Function

Base.pushfirst! — Function

Base.popfirst! — Function

Base.insert! — Function

Base.deleteat! — Function

Base.keepat! — Function

Base.splice! — Function

Base.resize! — Function

Base.append! — Function

Base.prepend! — Function

Core.Pair — Type

Base.Pairs — Type

Julia Notes Day1 -- functions and operators

v like — Wed, 13 Mar 2024 03:48:52 GMT

Functions

∑(n) = sum(1:n)

∑(3) # Output: 6
∑(100) # Output: 5050

Unicode variable and function names ∑ are allowed.

Julia functions can be defined using compact "assignment form" syntax.

The value returned by a function is the value of the last expression evaluated, the return keyword can be omited.

∏(n) = (p=1; for i in 1:n; p*=i; end; p) # the ; chain syntax

∏(4) # Output: 24
∏(10) # Output: 3628800

Sometimes it is convenient to have a single expression which evaluates several subexpressions in order, returning the value of the last subexpression as its value.

There are two Julia constructs that accomplish this: begin blocks and ; chains.

function foo(a,b)
    a+b, a*b #return multiple values by returning a tuple
end

foo(2,3) # Output: (5, 6)

Julia has a built-in data structure called a tuple that is closely related to function arguments and return values.

Operators

In Julia, operators and symbols often have specific meanings or uses that are integral to the language's syntax and functionality. Below is a list of some of these symbols, along with brief explanations and code examples to illustrate their use:

Arithmetic Operators

+ (Addition)
- (Subtraction)
* (Multiplication)
/ (Division)
÷ (integer divide)
^ (Power)
% (Remainder)

Comparison Operators

== (Equal to)
!= or ≠ (Not equal to)
< (Less than)
> (Greater than)
<= or ≤ (Less than or equal to)
>= or ≥ (Greater than or equal to)

Logical Operators

! (Not)
&& (And)
|| (Or)

Bitwise Operators

& (And)
| (Or)
~ (Not)
<< (logical/arithmetic shift left)
>> (arithmetic shift right)
>>> (logical shift right)
⊻ bitwise xor (exclusive or)
⊼ bitwise nand (not and)
⊽ bitwise nor (not or)

Spe cial Symbols

: (Colon, used for ranges)
; (Semicolon, used to separate expressions)
=> (Pair, used in dictionaries)
[] (Brackets, used for indexing or array literals)
{} (Curly braces, used for comprehension or generator expressions)
() (Parentheses, used for grouping or function calls)
|> (Pipe, used for function chaining)
. (Dot, used for broadcasting)

# Range
1:5  # 1:5

# Pair and Dictionary
dict = Dict("a" => 1, "b" => 2)

# Pipe
sqrt(25) |> println  # 5.0

# Broadcasting
[1, 2, 3] .+ 1  # [2, 3, 4]
1:3 .|> (x -> x^2)  # [1, 4, 9]

\ (Backslash, used for solving linear equations, A \ b solves Ax = b)
// (Rational, creates a rational number)

# Solving linear equations
A = [1 2; 3 4]
b = [5, 11]
A \ b  # [1.0, 2.0]

# Rational number
3 // 4  # 3//4

在 Python 中模拟 switch 语句的 fall-through 行为

v like — Mon, 05 Feb 2024 06:44:26 GMT

什么是 fall-through

在 JavaScript 中，switch 语句允许 fall-through 行为。当在 switch 语句中发现匹配条件时，相应的代码块将被执行，直到遇到 break 语句，此时 switch 语句将退出。如果没有提供 break 语句，程序将继续执行下一个 case，而不管它是否与条件匹配。这就是所谓的 fall-through 行为。

switch (Animal) {
    case "Dog":
    case "Cow":
        console.log("This animal is friend of people.");
    case "Giraffe":
    case "Pig":
        console.log("This animal is not extinct.");
        break;
    case "Dinosaur":
    default:
        console.log("This animal is extinct.");
}

fall-through 何时有用

在多个 cases 共享同一代码块或需要跨多个 cases 执行一连串操作时，switch 语句中的 fall-through 会非常有用。以下是一些例子：

将多个 cases 组合在一起：当多个 cases 需要执行相同的代码块时，fall-through 可让您一个接一个地列出这些 cases，而无需重复每个 case 的代码块。这可以使代码更简洁，更易于维护。

  switch (dayOfWeek) {
      case "Saturday":
      case "Sunday":
          console.log("It's the weekend!");
          break;
      default:
          console.log("Looking forward to the Weekend");
  }

不同 cases 下的顺序操作：在需要执行一系列操作的情况下，如果这些操作是建立在彼此基础之上的，但只是针对不同的情况从序列中的不同点开始执行，那么 fall-through 可以让您在不重复代码的情况下对这些操作进行结构化处理。

  switch (userLevel) {
      case "beginner":
          beginnerTasks();
          // Fall through to execute intermediate tasks as well.
      case "intermediate":
          intermediateTasks();
          // Fall through to execute advanced tasks as well.
      case "advanced":
          advancedTasks();
          break;
  }

设置特定条件下默认值：您可以使用 fall-through 功能为某些情况设置默认值，然后在后面的特定情况中覆盖这些默认值。

  switch (productType) {
      case "book":
          basePrice = 5;
          break;
      case "video":
      case "software":
          // These types have a base price and additional licensing fee.
          basePrice = 20;
          // Fall through to add licensing fee.
      default:
          licensingFee = 10;
  }

复杂的条件逻辑：当触发一个案例的逻辑比较复杂，涉及多个步骤或条件，并且这些步骤或条件相互依存时，可以使用 fall-through 功能以可读和高效的方式来构建这一逻辑。

  switch (true) {
      case (score >= 90):
          grade = "A";
          break;
      case (score >= 80):
          // Fall through to check for specific conditions within this range.
      case (score >= 70 && extraCredit):
          grade = "B";
          break;
      default:
          grade = "C";
  }

这些示例演示了在多个 cases 共享逻辑或需要跨 case 执行一系列操作的情况下，如何利用fall-through 来编写更高效、可读性和可维护性更强的代码。

Python 的模式匹配

Python 3.10 引入了一个名为 match-case 语句的新特性，也称为模式匹配。该特性为处理条件逻辑提供了一种更具表现力的方式，类似于其他语言（如 JavaScript 或 C++）中的 switch 语句。

然而，与 switch 语句不同的是，在 Python 的 match-case 语句中，cases 之间没有 "fall-through"。

下面是一个如何在 Python 中使用 match-case 语句的示例：

status = 404  # replace with the actual status
match status:
    case 400:
        result = "Bad request"
    case 404:
        result = "Not found"
    case 418:
        result = "I'm a teapot"
    case _:
        result = "Unknown status"
print(result)

在这个示例中，match 关键字后面是要匹配的表达式 (status)，case 后面是要匹配的模式。最后一个 case 语句中的 _ 是一个通配符，可以匹配任何内容，类似于 switch 语句中的 default case。

您还可以使用更复杂的模式，并在模式中添加 guard 子句。guard 是一个 if 子句，它进一步细化了 case 的条件。如果 guard 为 false，则 match 会继续尝试下一个 case 块。

point = (1, 1)  # replace with the actual point
match point:
    case (x, y) if x == y:
        # the case statement matches a tuple and 
        # uses a guard to check if the two elements of the tuple are equal.
        print(f"The point is located on the diagonal Y=X at {x}.")
    case (x, 0):
        print(f"X={x} and the point is on the x-axis.")
    case (0, y):
        print(f"Y={y} and the point is on the y-axis.")
    case (0, 0):
        print("The point is at the origin.")
    case (x, y) if x > 0 and y > 0:
        print(f"The point is in the first quadrant at ({x}, {y}).")
    case _:
        print(f"Point is not on the diagonal or axes at ({x}, {y}).")

如何在 Python 中模拟 switch 的 fall-through

不使用 fall-through

if user_level == "beginner":
    beginnerTasks()
    intermediateTasks()
    advancedTasks()
elif user_level == "intermediate":
    intermediateTasks()
    advancedTasks()
elif user_level == "advanced":
    advancedTasks()

模拟 fall-through

在 Python 中，您可以使用一系列 if 语句和一个标志变量来模拟 switch 语句的 fall-through 行为。以下代码可供参考：

fall_through = False

if user_level == "beginner":
    beginnerTasks()
    fall_through = True
if fall_through or user_level == "intermediate":
    intermediateTasks()
    fall_through = True
if fall_through or user_level == "advanced":
    advancedTasks()

在这段代码中，fall_through 变量用于控制是否执行后续 if 语句中的代码。如果 user_level 为 "beginner"，则调用 beginnerTasks()，并将 fall_through 设为 True。这将导致执行下一个 if 语句中的代码，而不验证 user_level 的值。同样的逻辑也适用于 "intermediate" 级别。如果 user_level 是 "advanced"，那么只有 advancedTasks() 被调用，因为 fall_through 在开始时是 False。

Python 的 match-case 语句的优点

在支持 fall-through 的语言（如 C 或 JavaScript）中，它的风险在于，如果程序员忘记在每个 case 的结尾加入 break 语句，就可能导致错误。如果没有 break，程序将继续执行下一个 case 的代码，这可能不是预期的行为，会导致意想不到的结果。

Python 3.10 引入的 match-case 语句的优点包括：

无跳转： Python 的 match-case 没有 fall-through 行为，这消除了意外执行多个 cases 的风险。一旦发现匹配，Python 会执行相应 case 块中的语句，然后跳到 match 块的末尾，继续执行程序的其余部分。
模式匹配：match-case 语句是为模式匹配而设计的，它比简单的值匹配更强大。它允许匹配类型模式，而不仅仅是值，这可以使代码更具表现力和可读性。
复杂匹配： Python 的 match-case 可以处理复杂的匹配，并且可以在单个 case 块中处理多种可能的情况，这可以简化逻辑并减少嵌套 if 语句的需要。
可读性： match-case 语句提供了一种结构清晰的方式来处理多个条件，从而使代码更具可读性，这通常比一长串 if-elif-else 语句更优雅。
多重模式：您可以使用 | 操作符在单例中匹配多个模式，从而以简洁的方式处理多个相关条件，而无需重复代码。
默认情况：match-case 语句中的默认情况用下划线 _ 表示，它可以捕获前面 cases 中没有匹配到的任何值，确保涵盖所有可能性。

总之，Python 的 match-case 语句为传统 fall-through 行为的 switch 语句提供了一个更安全、更强大的替代方案，减少了 bug 的风险，提高了代码的清晰度。

Mimic the fall-through behavior of switch in Python

v like — Mon, 05 Feb 2024 05:58:39 GMT

What is fall-through

In JavaScript, the switch statement allows for fall-through behavior. When a match is found in a switch statement, the corresponding block of code is executed until a break statement is encountered, at which point the switch statement is exited. If a break statement is not provided, the program will continue executing the next case, regardless of whether it matches the condition or not. This is known as fall-through behavior.

switch (Animal) {
    case "Dog":
    case "Cow":
        console.log("This animal is friend of people.");
    case "Giraffe":
    case "Pig":
        console.log("This animal is not extinct.");
        break;
    case "Dinosaur":
    default:
        console.log("This animal is extinct.");
}

When will fall-through be useful

Fall-through in a switch statement can be very useful in several scenarios where multiple cases share the same code block or when a sequence of operations needs to be executed across multiple cases. Here are some examples where fall-through is particularly beneficial:

Grouping Multiple Cases Together: When several cases should execute the same block of code, fall-through allows you to list these cases one after the other without repeating the code block for each case. This can make the code more concise and easier to maintain.
```
 switch (dayOfWeek) {
     case "Saturday":
     case "Sunday":
         console.log("It's the weekend!");
         break;
     default:
         console.log("Looking forward to the Weekend");
 }
```

Sequential Operations for Different Cases: In situations where you want to execute a sequence of operations that build on each other, but only start at different points in the sequence for different cases, fall-through allows you to structure these operations without duplicating code.

 switch (userLevel) {
     case "beginner":
         beginnerTasks();
         // Fall through to execute intermediate tasks as well.
     case "intermediate":
         intermediateTasks();
         // Fall through to execute advanced tasks as well.
     case "advanced":
         advancedTasks();
         break;
 }

Setting Defaults with Overridden Specifics: You can use fall-through to set default values for certain cases, then override those defaults for specific cases that follow.

 switch (productType) {
     case "book":
         basePrice = 5;
         break;
     case "video":
     case "software":
         // These types have a base price and additional licensing fee.
         basePrice = 20;
         // Fall through to add licensing fee.
     default:
         licensingFee = 10;
 }

Complex Conditional Logic: When the logic for triggering a case is complex and involves multiple steps or conditions that build upon each other, fall-through can be used to structure this logic in a readable and efficient manner.

 switch (true) {
     case (score >= 90):
         grade = "A";
         break;
     case (score >= 80):
         // Fall through to check for specific conditions within this range.
     case (score >= 70 && extraCredit):
         grade = "B";
         break;
     default:
         grade = "C";
 }

These examples illustrate how fall-through can be leveraged to write more efficient, readable, and maintainable code in scenarios where multiple cases share logic or when a sequence of operations needs to be executed across cases.

Python's pattern matching

Python 3.10 introduced a new feature called the match-case statement, also known as pattern matching. This feature provides a more expressive way to handle conditional logic and is similar to the switch statement found in other languages like JavaScript or C++.

However, unlike the switch statement, there is no "fall-through" between cases in Python's match-case statement.

Here's an example of how to use the match-case statement in Python:

status = 404  # replace with the actual status
match status:
    case 400:
        result = "Bad request"
    case 404:
        result = "Not found"
    case 418:
        result = "I'm a teapot"
    case _:
        result = "Unknown status"
print(result)

In this example, the match keyword is followed by the expression to match (status), and case is followed by the pattern to match against. The _ in the last case statement is a wildcard that matches anything, similar to the default case in a switch statement.

You can also use more complex patterns and add a guard clause to a pattern. A guard is an if clause that further refines the condition for the case. If the guard is false, match goes on to try the next case block.

point = (1, 1)  # replace with the actual point
match point:
    case (x, y) if x == y:
        # the case statement matches a tuple and 
        # uses a guard to check if the two elements of the tuple are equal.
        print(f"The point is located on the diagonal Y=X at {x}.")
    case (x, 0):
        print(f"X={x} and the point is on the x-axis.")
    case (0, y):
        print(f"Y={y} and the point is on the y-axis.")
    case (0, 0):
        print("The point is at the origin.")
    case (x, y) if x > 0 and y > 0:
        print(f"The point is in the first quadrant at ({x}, {y}).")
    case _:
        print(f"Point is not on the diagonal or axes at ({x}, {y}).")

How to mimic the fall-through behavior of switch in Python

Without fall-through

if user_level == "beginner":
    beginnerTasks()
    intermediateTasks()
    advancedTasks()
elif user_level == "intermediate":
    intermediateTasks()
    advancedTasks()
elif user_level == "advanced":
    advancedTasks()

Mimic the fall-through

You can mimic the fall-through behavior of a switch statement in Python using a series of if statements and a flag variable. Here's the code for reference:

fall_through = False

if user_level == "beginner":
    beginnerTasks()
    fall_through = True
if fall_through or user_level == "intermediate":
    intermediateTasks()
    fall_through = True
if fall_through or user_level == "advanced":
    advancedTasks()

In this code, the fall_through variable is used to control whether the code in the subsequent if statements should be executed. If user_level is "beginner", then beginnerTasks() is called and fall_through is set to True. This causes the code in the next if statement to be executed, regardless of the value of user_level. The same logic applies to the "intermediate" level. If user_level is "advanced", then only advancedTasks() is called, because fall_through is False at the start.

The pros of Python's `match-case` statement

The risk of fall-through in languages that support it, such as C or JavaScript, is that it can lead to bugs if the programmer forgets to include a break statement at the end of each case. Without a break, the program will continue executing the next case's code, which might not be the intended behavior and can cause unexpected results.

The pros of Python's match-case statement, introduced in Python 3.10, include:

No Fall-Through: Python's match-case does not have fall-through behavior, which eliminates the risk of accidentally executing multiple cases. Once a match is found, Python executes the statements in the corresponding case block and then skips to the end of the match block, continuing with the rest of the program.
Pattern Matching: The match-case statement is designed for pattern matching, which is more powerful than simple value matching. It allows for matching patterns of types, not just values, which can lead to more expressive and readable code.
Complex Matches: Python's match-case can handle complex matches and can be designed to handle multiple possible cases in a single case block, which can simplify the logic and reduce the need for nested if statements.
Readability: The match-case statement can make code more readable by providing a clear and structured way to handle multiple conditions, which is often more elegant than a long series of if-elif-else statements.
Multiple Patterns: You can match multiple patterns in a single case using the | operator, which allows for a concise way to handle multiple related conditions without duplicating code.
Default Case: The default case in a match-case statement is represented by an underscore _, which acts as a catch-all for any values not matched by previous cases, ensuring that all possibilities are covered.

In summary, Python's match-case statement provides a safer and more powerful alternative to traditional switch statements with fall-through behavior, reducing the risk of bugs and improving code clarity.

防范并识别 html/JS 应用内存泄漏的策略

v like — Wed, 24 Jan 2024 10:41:51 GMT

随着时间的推移，内存泄漏会降低应用程序的性能，并可能导致崩溃或其他问题。因此，在应用程序中有效管理内存非常重要。

防范内存泄漏

要防范 HTML/JS 应用程序中的内存泄漏，您可以遵循以下策略：

避免不必要的引用：不必要的引用是内存泄漏的常见原因。当不再需要的对象仍被代码的其他部分引用时，就会发生这种情况，从而导致垃圾回收器无法释放这些对象所使用的内存[3]。
```
 var detachedNode = document.getElementById('someElement');
 detachedNode.parentNode.removeChild(detachedNode);
```
在上面的示例中，尽管 someElement 已从 DOM 中移除，但由于 detachedNode 中存在对它的引用，所以它仍在内存中。
合适的事件监听器管理：不删除事件监听器会导致内存泄漏。当不再需要事件监听器时，请务必将其移除。
避免使用全局变量：全局变量不会被垃圾回收器回收，因为它们处于全局范围内。如果全局变量持有对某个对象的引用，则该对象无法被垃圾回收器回收[1]。
避免闭包引用不需要的数据：闭包可能会无意中保留对不再需要的较大对象或作用域的引用，从而导致内存泄漏[1]。
```
  function outerFunction() {
      var largeObject = new Array(1000000).fill('Memory Leak');
      return function innerFunction() {
          return true;
      }
  }
  var closure = outerFunction();
```
在上例中，innerFunction 的闭包存在于 outerFunction 的作用域里，其中包括 largeObject。尽管 innerFunction 不会使用 largeObject，但由于闭包的存在，只要 innerFunction 存在，它就会保留在内存中。
避免第三方库的内存泄漏：某些第三方库可能存在内存泄漏。请始终使用最新版本的库和框架，因为这些版本通常都有内存泄漏修复[1]。
代码审查：定期审查代码有助于捕捉常见的内存泄漏模式[1]。
删除不必要的 DOM 元素：如果您动态创建了 DOM 元素，但以后不再需要它们，请确保删除它们以及对它们的任何引用[5]。

识别内存泄漏

要识别 HTML/JS 应用程序中的内存泄漏，可以使用以下工具和库：

MemLab： MemLab 是由 Meta（前身为 Facebook）开发的一个开源框架，用于查找 JavaScript 内存泄漏。它能自动检测内存泄漏，并结合特定框架知识来完善泄漏对象列表。MemLab 可以生成驻留追踪（retainer traces），帮助开发人员了解对象未被垃圾回收的原因[6]。
Chrome 开发者工具： Chrome 开发者工具（DevTools）提供了一系列内存使用情况剖析工具。时间轴视图和配置文件视图（现在是新版 Chrome 浏览器内存面板的一部分）对于发现异常内存模式和识别内存泄漏至关重要。时间轴视图可以显示内存使用量的周期性跳变，而这种跳变在垃圾回收后并不会缩小，从而帮助你发现内存泄漏。分配时间轴/剖析器会周期性拍摄堆快照，帮助你研究可疑的内存分配图[4][7]。

这些工具可以深入了解内存分配和驻留情况，帮助你检测内存泄漏，从而优化代码，提高性能和稳定性。

参考链接

[1] https://nolanlawson.com/2020/02/19/fixing-memory-leaks-in-web-applications/

[2] https://www.ditdot.hr/en/causes-of-memory-leaks-in-javascript-and-how-to-avoid-them

[3] https://blog.logrocket.com/escape-memory-leaks-javascript/

[4] https://auth0.com/blog/four-types-of-leaks-in-your-javascript-code-and-how-to-get-rid-of-them/

[5] https://stackoverflow.com/questions/49147182/deleting-html-elements-in-javascript-to-avoid-memory-leaks

[6] https://engineering.fb.com/2022/09/12/open-source/memlab/

[7] https://www.lambdatest.com/blog/eradicating-memory-leaks-in-javascript/

Strategies to prevent/identify memory leaks in HTML/JS

v like — Wed, 24 Jan 2024 10:15:40 GMT

Memory leaks can degrade the performance of your application over time and can lead to crashes or other issues. Therefore, it's important to manage memory effectively in your applications.

Prevent Memory Leaks

To prevent memory leaks in HTML/JS applications, you can follow these strategies:

Avoid Unwanted References: Unwanted references are a common cause of memory leaks. These occur when objects that are no longer needed are still referenced by other parts of your code, preventing the garbage collector from freeing up the memory used by these objects[3].
```
 var detachedNode = document.getElementById('someElement');
 detachedNode.parentNode.removeChild(detachedNode);
```
In the above example, even though someElement has been removed from the DOM, it's still in memory because there's a reference to it in detachedNode.
Proper Event Listener Management: Not removing event listeners can lead to memory leaks. Always remove event listeners when they are no longer needed.
Avoid Global Variables: Global variables are not garbage collected as they are in the global scope. If a global variable holds a reference to an object, that object cannot be garbage collected[1].
Avoid Closures that Reference Unneeded Data: Closures can inadvertently keep references to larger objects or scopes that are no longer needed, leading to memory leaks[1].
```
 function outerFunction() {
     var largeObject = new Array(1000000).fill('Memory Leak');
     return function innerFunction() {
         return true;
     }
 }
 var closure = outerFunction();
```
In the above example, innerFunction has a closure over the scope of outerFunction, which includes largeObject. Even though largeObject is not used by innerFunction, it's kept in memory as long as innerFunction exists because of the closure.
Avoid Memory Leaks in Third-Party Libraries: Some third-party libraries may have memory leaks. Always use the latest version of libraries and frameworks, as these versions often have memory leak fixes[1].
Code Review: Regular code reviews can help catch common memory leak patterns[1].
Delete Unnecessary DOM Elements: If you create DOM elements dynamically and don't need them later, make sure to remove them and any references to them[5].

Identify Memory Leaks

To identify memory leaks in HTML/JS applications, you can use the following tools and libraries:

MemLab: MemLab is an open-source framework developed by Meta (formerly Facebook) for finding JavaScript memory leaks. It automates the detection of memory leaks and incorporates framework-specific knowledge to refine the list of leaked objects. MemLab can generate retainer traces to help developers understand why objects are not being garbage collected[6].
Chrome DevTools: Chrome DevTools provides a set of tools to profile memory usage. The Timeline view and the Profiles view (now part of the Memory panel in newer versions of Chrome) are essential for discovering unusual memory patterns and identifying memory leaks. The Timeline view can help you spot big leaks by showing periodic jumps in memory usage that do not shrink after garbage collection. The Allocation Timeline/Profiler takes heap snapshots periodically and helps you study the graph for suspicious memory allocation[4][7].

These tools can help you detect memory leaks by providing insights into memory allocation and retention, allowing you to optimize your code for better performance and stability.

Citations:

[1] https://nolanlawson.com/2020/02/19/fixing-memory-leaks-in-web-applications/

[2] https://www.ditdot.hr/en/causes-of-memory-leaks-in-javascript-and-how-to-avoid-them

[3] https://blog.logrocket.com/escape-memory-leaks-javascript/

[4] https://auth0.com/blog/four-types-of-leaks-in-your-javascript-code-and-how-to-get-rid-of-them/

[5] https://stackoverflow.com/questions/49147182/deleting-html-elements-in-javascript-to-avoid-memory-leaks

[6] https://engineering.fb.com/2022/09/12/open-source/memlab/

[7] https://www.lambdatest.com/blog/eradicating-memory-leaks-in-javascript/

Js 不导致栈溢出的递归函数

v like — Mon, 22 Jan 2024 09:23:03 GMT

堆栈溢出

递归函数（Recursive Function）如果调用自身次数过多，超过栈大小限制，就会导致栈溢出。

一个典型的例子是，用递归函数简陋实现斐波那契数列，如果调用参数过大，将导致栈溢出：

function fibonacci(n) {
  if (n < 2) return n;
  return fibonacci(n - 1) + fibonacci(n - 2);
}

for (let n = 0; n < 100; n++) {
  console.log(`fibo ${n}: ${fibonacci(n)}`)
}

这种用递归函数简陋实现的斐波那契数列可能由于大量的冗余计算导致程序运行缓慢。如果输入 n 过大，可能会超过最大调用栈大小，导致栈溢出。

异步递归

不过，并非所有多次调用自身的递归函数都会导致栈溢出。

在 JavaScript 中，当每次递归调用都被放在当前执行上下文完成后运行，就会发生异步递归。例如使用 setTimeout、setInterval 或异步函数（Promise/async/await）时。这里是一个使用 setTimeout 的示例：

function asyncRecursiveFunction(n) {
  if (n > 0) {
    console.log(n);
    setTimeout(() => asyncRecursiveFunction(n - 1), 1000);
  } else {
    console.log('Done');
  }
}

asyncRecursiveFunction(5);
/*
5
4
3
2
1
Done
*/

下面是另一个使用 Promise 的示例：

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

function sleepList(ts1, ts2, sleepID = 0) {
  if (ts1 >= ts2) {
    console.log('Finished processing array');
    return;
  }

  console.log(`sleep ${sleepID} begin ${ts1}ms`);

  sleep(ts1)
    .then(s => {
      console.log(`sleep ${sleepID} end ${ts1}ms`);
      sleepList(ts1 + 1, ts2, sleepID + 1);
    });
}

sleepList(1000, 1005);
/*
sleep 0 begin 1000ms
sleep 0 end 1000ms
sleep 1 begin 1001ms
sleep 1 end 1001ms
sleep 2 begin 1002ms
sleep 2 end 1002ms
sleep 3 begin 1003ms
sleep 3 end 1003ms
sleep 4 begin 1004ms
sleep 4 end 1004ms
Finished processing array
*/

尾递归优化 (Tail Call Optimization，TCO)

尾递归优化是指一个函数的最后一个操作是调用另一个函数（或自身），而当前函数的栈帧可以被新函数的栈帧所替代。这可以防止每次递归调用都导致栈增长。不同的编程语言及其编译器对尾递归优化的支持各不相同。当启用优化标记时，某些编译器（如 GCC）会在某些场景下消除尾部调用。

截至 2023 年，JavaScript 环境尚未广泛支持尾递归优化。

https://stackoverflow.com/questions/37224520/are-functions-in-javascript-tail-call-optimized

https://stackoverflow.com/questions/54719548/tail-call-optimization-implementation-in-javascript-engines

https://www.reddit.com/r/javascript/comments/pwwbky/askjs_why_so_little_support_for_tco_tail_call/

Recursive functions don't casue stack overflow in JS

v like — Mon, 22 Jan 2024 08:56:35 GMT

Stack Overflow

Recursive functions can cause a stack overflow if they call themselves too many times and exceed the stack size limit.

A classic example of a recursive function that can exceed the stack size limit if called with a large input is the naive implementation of the Fibonacci sequence:

function fibonacci(n) {
  if (n < 2) return n;
  return fibonacci(n - 1) + fibonacci(n - 2);
}

for (let n = 0; n < 100; n++) {
  console.log(`fibo ${n}: ${fibonacci(n)}`)
}

This naive implementation of the Fibonacci sequence can cause a program to run slowly due to the large number of redundant computations. If the input n is too large, it can exceed the maximum call stack size and cause a stack overflow.

Asynchronous Recursion

However, not all recursive functions that call themselves multiple times will causing a stack overflow.

In JavaScript, asynchronous recursion can occur when each recursive call is scheduled to run after the current execution context has completed, such as when using setTimeout, setInterval, or asynchronous functions (Promise/async/await). Here's an example using setTimeout:

function asyncRecursiveFunction(n) {
  if (n > 0) {
    console.log(n);
    setTimeout(() => asyncRecursiveFunction(n - 1), 1000);
  } else {
    console.log('Done');
  }
}

asyncRecursiveFunction(5);
/*
5
4
3
2
1
Done
*/

Here's another example using Promise:

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

function sleepList(ts1, ts2, sleepID = 0) {
  if (ts1 >= ts2) {
    console.log('Finished processing array');
    return;
  }

  console.log(`sleep ${sleepID} begin ${ts1}ms`);

  sleep(ts1)
    .then(s => {
      console.log(`sleep ${sleepID} end ${ts1}ms`);
      sleepList(ts1 + 1, ts2, sleepID + 1);
    });
}

sleepList(1000, 1005);
/*
sleep 0 begin 1000ms
sleep 0 end 1000ms
sleep 1 begin 1001ms
sleep 1 end 1001ms
sleep 2 begin 1002ms
sleep 2 end 1002ms
sleep 3 begin 1003ms
sleep 3 end 1003ms
sleep 4 begin 1004ms
sleep 4 end 1004ms
Finished processing array
*/

Tail Call Optimization (TCO)

Tail call optimization is a feature where the last action of a function is a call to another function (or itself), and the current function's stack frame can be replaced with the new function's stack frame. This prevents the stack from growing with each recursive call. The support for Tail Call Optimization (TCO) varies across different programming languages and their compilers. Some compilers, such as GCC, have been known to perform tail call elimination in certain cases when optimization flags are used.

As of 2023, TCO is not widely supported in JavaScript environments.

https://stackoverflow.com/questions/37224520/are-functions-in-javascript-tail-call-optimized

https://stackoverflow.com/questions/54719548/tail-call-optimization-implementation-in-javascript-engines

https://www.reddit.com/r/javascript/comments/pwwbky/askjs_why_so_little_support_for_tco_tail_call/

Node.js 从列表中查找最快的 HTTP/2 DoH 服务器

v like — Fri, 19 Jan 2024 13:05:30 GMT

对 DoH（DNS over HTTPS）来说 HTTP/2 尤其有效，因为 HTTP/2 支持多路复用，允许在同一连接上发送多个 DNS 查询，而无需等待之前的响应。这可以大大提高 DNS 解析的性能，尤其是在同时进行多个查询时。

网络上有数百个公开可用的 DoH 服务器，我们可以使用 Node.js HTTP/2 模块来查找在一定阈值（如 1 秒）内响应最快的服务器。

首先，让我们定义一个函数 check_url，它将返回一个 promise，并在 HTTP/2 请求成功时转为 resolve 状态。

const http2 = require('http2');

const timeoutDuration = 1000; // 1 seconds

function check_url(URL) {

    let fnResolve, fnReject;
    let myPromise = new Promise((resolve, reject) => {

        fnResolve = resolve;
        fnReject = reject;
    });

    const client = http2.connect(URL);//https://dns.alidns.com , https://dns.quad9.net

    const req = client.request({ ':path': '/dns-query?dns=AAABAAABAAAAAAAAA3d3dwdleGFtcGxlA2NvbQAAAQAB' });

    req.on('response', (headers, flags) => {
        for (const name in headers) {
            console.log(`${name}: ${headers[name]}`);
        }

        fnResolve('    ' + URL);
    });

    return myPromise;
}

让我们添加一些处理逻辑，当错误或超时发生时将 promise 转为 reject 状态。

    req.on('error', (err) => {
        console.error('ClientHttp2Stream error: ' + err);
        req.close(http2.constants.NGHTTP2_CANCEL);

        fnReject('!!! ' + URL)
    });

    // Set a timeout on the request
    req.setTimeout(timeoutDuration, () => {
        console.error(`ClientHttp2Stream ${URL} timed out`);

        req.destroy();

        fnReject('--- ' + URL)
    });

最后，添加一个处理函数，在请求流完成并关闭时同时关闭 HTTP/2 连接（请记住，HTTP/2 允许在同一连接上进行多个并发流，因此连接不会自动关闭）。

req.on('close', (e) => {
    console.log( 'ClientHttp2Stream is closed' )
    client.close();
})

现在，我们可以使用递归函数逐个检查列表中的 url，并返回响应时间在 1 秒以内的最快服务器。

let dohs = ["https://dns.google/dns-query",
    "https://cloudflare-dns.com/dns-query",
    "https://dns.alidns.com/dns-query",
    "https://dns.quad9.net/dns-query"];

/*
* check_next check doh[index], then doh[++index]...
* show: 
*      0 show urls response within timeoutDuration
*      1 show urls response great than timeoutDuration
*      2 don't show urls, for debug
*/
function check_next(index = 0, { show = 0 } = {}) {
    if (index >= dohs.length || index < 0) return;

    check_url(dohs[index])
        .then(URL => { if (0 === show) console.log(URL) })
        .catch(error => { if (1 === show) console.log(error) })
        .finally(() => {
            //only check_next when last check is done
            check_next(++index, { show })
        });
}

check_next(0, { show: 0 })

运行 Node 代码，获取最快服务器列表

node find-fastest-doh-servers.js
# 输出结果
#    https://dns.alidns.com/dns-query
#    https://dns.quad9.net/dns-query

完整代码在 GitHub Gist 上面

https://gist.github.com/likev/7c26cec0103b3187ddcb793f94ae593c

Node.js Find Fastest HTTP/2 Servers from DoH List

v like — Fri, 19 Jan 2024 12:49:08 GMT

HTTP/2 is particularly beneficial for DoH(DNS over HTTPS) because it allows multiple DNS queries to be sent over the same connection without waiting for previous responses, thanks to its support for multiplexing. This can significantly improve the performance of DNS resolutions, especially when multiple queries are being made.

When hundreds of servers are publicly available, we can use the Node.js HTTP/2 module to find the fastest ones that respond within duration (such as 1s).

First, let's define a function check_url that returns a promise and resolves if the HTTP/2 request is successful.

const http2 = require('http2');

const timeoutDuration = 1000; // 1 seconds

function check_url(URL) {

    let fnResolve, fnReject;
    let myPromise = new Promise((resolve, reject) => {

        fnResolve = resolve;
        fnReject = reject;
    });

    const client = http2.connect(URL);//https://dns.alidns.com , https://dns.quad9.net

    const req = client.request({ ':path': '/dns-query?dns=AAABAAABAAAAAAAAA3d3dwdleGFtcGxlA2NvbQAAAQAB' });

    req.on('response', (headers, flags) => {
        for (const name in headers) {
            console.log(`${name}: ${headers[name]}`);
        }

        fnResolve('    ' + URL);
    });

    return myPromise;
}

Let's also add logic to reject the promise if an error or timeout occurs.

    req.on('error', (err) => {
        console.error('ClientHttp2Stream error: ' + err);
        req.close(http2.constants.NGHTTP2_CANCEL);

        fnReject('!!! ' + URL)
    });

    // Set a timeout on the request
    req.setTimeout(timeoutDuration, () => {
        console.error(`ClientHttp2Stream ${URL} timed out`);

        req.destroy();

        fnReject('--- ' + URL)
    });

Finally, add a deal function to close the HTTP/2 connection when the request stream is complete and close (remember that HTTP/2 allows multiple concurrent exchanges on the same connection, so the connection will not close automatically).

req.on('close', (e) => {
    console.log( 'ClientHttp2Stream is closed' )
    client.close();
})

Now we can use a recursive function to check urls from a list one by one and return the fastest server that responds under 1s.

let dohs = ["https://dns.google/dns-query",
    "https://cloudflare-dns.com/dns-query",
    "https://dns.alidns.com/dns-query",
    "https://dns.quad9.net/dns-query"];

/*
* check_next check doh[index], then doh[++index]...
* show: 
*      0 show urls response within timeoutDuration
*      1 show urls response great than timeoutDuration
*      2 don't show urls, for debug
*/
function check_next(index = 0, { show = 0 } = {}) {
    if (index >= dohs.length || index < 0) return;

    check_url(dohs[index])
        .then(URL => { if (0 === show) console.log(URL) })
        .catch(error => { if (1 === show) console.log(error) })
        .finally(() => {
            //only check_next when last check is done
            check_next(++index, { show })
        });
}

check_next(0, { show: 0 })

Run node code and get the fastest list

node find-fastest-doh-servers.js
# output
#    https://dns.alidns.com/dns-query
#    https://dns.quad9.net/dns-query

The full code can be found on GitHub Gist

https://gist.github.com/likev/7c26cec0103b3187ddcb793f94ae593c