Comments (2)
@HoustonPutman
The same issue is happening to me,
The HPA mentioned it would scale to 14 pods, but they kept 58 running.
For me the leader election was successful, but I could see a lot of down replicas which is causing query issues and getting the error like shards are down
from solr-operator.
Ok, so y'alls issues seem somewhat related.
I have seen problems with Solr failing to delete bad replicas during an unsuccessful migration. And that's the reason why you are seeing a large increase in the number of replicas.
So i suspect something wrong with the scale down/up / migration of the shards. Every pod gets restarted during the downgrade......
This is definitely a problem, and related to the fact that you are addressing your solr nodes through the ingress. In order for all Solr traffic to not be directed through the ingress (which would slow things down considerably), we use basically /etc/hosts on the pods to map each ingress address to the IP of the pod it maps to. And since you are scaling down, it is removing some of the /etc/hosts entries, thus requiring full restarts every time.
An easy solution to this would be to only update the /etc/hosts if an IP is changed or added. It doesn't really matter if we have unused entries there.
Anyways, we should definitely have an integration test that stresses the HPA with ingresses, because this seems like a very iffy edge case.
The same issue is happening to me
@sabaribose I think this is separate, because you are not using an ingress, but using the headless service.
I think your is from the BalanceReplicas command not queueing for a retry when it fails. But I will do more investigation here.
from solr-operator.
Related Issues (20)
- Support replicaPlacementFactory in solr.xml HOT 2
- Liveness probe failing for Prometheus Exporter connected to a large SolrCloud
- Disabling PodDisruptionBudgets for both zk pods and solr pods HOT 3
- adding automountServiceAccountToken HOT 1
- Replica allocation after Node is DisabledScheduling HOT 1
- zkHost and zkServer generated incorrectly - helm templates HOT 2
- Solr 8.11 with SolrMetrics produces duplicate samples with prometheus v2.52 HOT 12
- Scale down operation fails and is never requeued if `getReplicasForPod` fails transiently HOT 2
- Configure Resources for zookeeper operator HOT 1
- Allow resizing (expanding) of persistent data PVCs
- Upgrade from Kubebuilder 3 to 4
- SolrOperator leads to 404 HOT 1
- Facing trouble while restoring solr in 8.11.3 HOT 1
- Job Solr-operator-zookeeper-operator-pre-delete without limits
- Unable to pass shareProcessNamespace to PodOptions
- It is impossible to setup TLS between Solr and Zookeeper HOT 1
- Cannot specify an imagePullSecret for solr-operator Helm chart
- Configuration of Solr MultiAuthplugin with JWT and basic auth gives the error of PKI authentication on creating cores. HOT 9
- [Regression] security.json is not uploaded during the first initialization of SolrCloud HOT 1
- How to keep the configsets directiory in solr pods with deployed with helm chart
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from solr-operator.